For decades,\u00a0Reinforcement Learning (RL)\u00a0has been the driving force behind breakthroughs in\u00a0robotics, game-playing AI (AlphaGo, OpenAI Five), and control systems. RL\u2019s strength lies in its ability to optimize decision-making by\u00a0maximizing long-term rewards, making it ideal for problems requiring sequential reasoning. However,\u00a0large language models (LLMs)\u00a0initially relied on\u00a0supervised learning, where models were fine-tuned on static datasets. This approach lacked adaptability\u2014while LLMs could mimic human text, they struggled with nuanced human preference alignment, leading to inconsistencies in conversational AI. The introduction of\u00a0RLHF (Reinforcement Learning with Human Feedback)\u00a0changed everything. By integrating RL into LLM fine-tuning, models like\u00a0ChatGPT, DeepSeek, Gemini, and Claude\u00a0could optimize (LLM Optimization) their responses based on user feedback. <\/p>\n
However, standard\u00a0PPO-based RLHF\u00a0had inefficiencies, requiring expensive reward modeling and iterative training. Enter\u00a0DeepSeek\u2019s Group Relative Policy Optimization (GRPO)\u2014a breakthrough that\u00a0eliminated the need for explicit reward modeling\u00a0by directly optimizing\u00a0preference rankings. To fully grasp the significance of GRPO, we must first explore the\u00a0fundamental policy optimization techniques (LLM optimization)\u00a0that power modern reinforcement learning.<\/p>\n
Learning Objectives\u00a0<\/h3>\n\n- Understand why RL-based techniques are crucial for optimizing LLMs like ChatGPT, DeepSeek, Claude, and Gemini.<\/li>\n
- Learn the fundamentals of policy optimization, including PG, TRPO, and PPO.Explore DPO and GRPO for preference-based LLM training without explicit reward models.<\/li>\n
- Compare PG, TRPO, PPO, DPO, and GRPO to determine the best approach for RL and LLM fine-tuning.<\/li>\n
- Gain hands-on experience with Python implementations of policy optimization algorithms.<\/li>\n
- Evaluate fine-tuning impact using training loss curves and probability distributions.<\/li>\n
- Apply DPO and GRPO to enhance LLM safety, alignment, and reliability.<\/li>\n<\/ul>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\nPrimer on Policy Optimization Techniques<\/h2>\n
Before diving into DeepSeek\u2019s GRPO, it\u2019s crucial to understand the policy optimization techniques that form the foundation of reinforcement learning (RL) in both traditional control tasks and LLM fine-tuning<\/a>. Policy optimization refers to the process of improving an AI agent\u2019s decision-making strategy (policy) to maximize expected rewards. While early methods like vanilla policy gradient (PG)<\/a> laid the groundwork, more sophisticated techniques such as TRPO, PPO, DPO, and GRPO evolved to address issues like stability, efficiency, and preference alignment.<\/p>\nWhat is Policy Optimization?<\/h3>\n
\u00a0 At its core, policy optimization is about learning the optimal policy \u03c0_\u03b8(a\u2223s)<\/b>, which maps a state s<\/i>\u00a0to an action a<\/i>\u00a0while maximizing long-term rewards. The objective function in RL is typically formulated as:\u00a0\u00a0<\/p>\n![\"Formula\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/Eq1.webp\")
<\/figure>\nWhere R(\u03c4)<\/b>\u00a0is the total reward collected in a trajectory \u03c4, and the expectation is taken over all possible trajectories following policy \u03c0_\u03b8<\/b>.<\/p>\n
There are three major approaches to policy optimization:<\/p>\n
1.\u00a0Gradient-Based Optimization (Policy Gradient Methods)<\/h4>\n\n- These methods directly compute gradients of expected reward and update policy parameters using gradient ascent.<\/li>\n
- Example: REINFORCE algorithm (Vanilla Policy Gradient).<\/li>\n
- Pros: Simple, works with continuous and discrete actions.<\/li>\n
- Cons: High variance, requires tricks like baseline subtraction.<\/li>\n<\/ul>\n
2.\u00a0Trust-Region Optimization (TRPO, PPO)<\/h4>\n\n- Introduces constraints (KL divergence) to ensure policy updates are stable and not too drastic.<\/li>\n
- Example: TRPO ensures updates stay within a \u201ctrust region\u201d; PPO simplifies this with clipping.<\/li>\n
- Pros: More stable than raw policy gradients.<\/li>\n
- Cons: Computationally expensive (TRPO), hyperparameter-sensitive (PPO).<\/li>\n<\/ul>\n
3.\u00a0Preference-Based Optimization (DPO, GRPO)<\/h4>\n\n- Optimizes directly from ranked human preferences instead of rewards.<\/li>\n
- Example: DPO learns from preferred vs. rejected responses; GRPO generalizes to groups.<\/li>\n
- Pros: Eliminates the need for reward models and better aligns LLMs with human intent.<\/li>\n
- Cons: Requires high-quality preference data.<\/li>\n<\/ul>\n
Mathematical Foundations (Required for All Methods)<\/h2>\nA. Markov Decision Process (MDP)<\/h3>\n
RL is typically formulated as a Markov Decision Process<\/b> (MDP), represented as:<\/p>\n![\"Formula\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/MDP.webp\")
<\/figure>\nwhere:<\/p>\n
\n- S<\/b> is the state space,<\/li>\n
- A<\/b> is the action space,<\/li>\n
- P(s\u2032\u2223s,a)<\/b> is the transition probability to state s\u2032,<\/li>\n
- R(s,a)<\/b> is the reward function,<\/li>\n
- \u03b3<\/b> is the discount factor (how much future rewards are valued).<\/li>\n<\/ul>\n
\u00a0 B. Expected Return J(\u03b8)<\/h3>\n
\u00a0 The Expected Return<\/b> (ER) measures how much cumulative reward we expect from following policy \u03c0_\u03b8<\/b>:\u00a0\u00a0<\/p>\n![\"Formula\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/ER.webp\")
<\/figure>\n\u00a0 where \u03b3<\/b> (0 \u2264 \u03b3<\/b> \u2264 1) determines how much future rewards contribute.\u00a0\u00a0<\/p>\nC. Policy Gradient Theorem<\/h3>\n
Policy gradient (PG) methods update the policy using gradients of expected rewards. The key equation:<\/p>\n![\"Formula\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PG.webp\")
<\/figure>\nwhere:<\/p>\n
\n- A(s,a)<\/b> is the advantage function (how good action a<\/b> is compared to average actions in state s<\/b>).<\/li>\n
- log\u03c0_\u03b8\u200b<\/b> ensures we increase the probabilities of better actions.<\/li>\n<\/ul>\n
D. Advantage Function A(s,a)<\/h3>\n
To reduce variance in gradient estimates, we use the advantage function:<\/p>\n![\"Formula\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/AdvFunc.webp\")
<\/figure>\nwhere:<\/p>\n
\n- Q(s,a)<\/b> is the expected return for taking action a<\/b> at state s<\/b>.<\/li>\n
- V(s) <\/b>is the expected return following policy \u03c0<\/b> from s<\/b>.<\/li>\n<\/ul>\n
Using A(s,a)<\/b>\u00a0helps make updates more stable and efficient.<\/p>\nPolicy Gradient (PG) \u2013 The Foundation<\/h2>\n
The Policy Gradient (PG) method is the most fundamental approach to reinforcement learning. Instead of learning a value function, PG directly parameterizes the policy \u03c0_\u03b8(a\u2223s)\u00a0<\/b>and updates it using gradient ascent. This allows learning in continuous action spaces, making it effective for tasks like robotics, game AI, and LLM fine-tuning.<\/p>\n
However, PG methods suffer from high variance due to their reliance on sampling full trajectories. More advanced methods like TRPO, PPO, and GRPO build upon PG to improve stability.<\/p>\n
The Policy Gradient Theorem<\/h2>\n
\u00a0 The goal of policy optimization is to find policy parameters \u03b8<\/b>\u00a0that maximize expected return:\u00a0\u00a0<\/p>\n![\"The](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PG_1.webp\")
<\/figure>\nUsing the log-derivative trick, we obtain the Policy Gradient Theorem:<\/p>\n![\"The](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PG_2.webp\")
<\/figure>\nwhere:<\/p>\n
\n- \u2207\u03b8\u200blog\u03c0\u03b8\u200b(a\u2223s) <\/b>is the gradient of the log-probability of taking action aaa.<\/li>\n
- A(s,a)<\/b> (Advantage function) determines how much better action aaa is compared to others.<\/li>\n
- We perform gradient ascent to increase the probability of good actions.<\/li>\n<\/ul>\n
Code Example: REINFORCE Algorithm<\/h2>\n
The REINFORCE algorithm is the simplest form of PG. It samples trajectories, computes rewards, and updates the policy parameters. Below is the main training loop (only the key function is shown to limit the scope; the full notebook is linked<\/a>).<\/p>\ndef train_policy_gradient(env, policy, optimizer, num_episodes=500, gamma=0.99):\n \"\"\"Train a policy using the REINFORCE algorithm\"\"\"\n reward_history = []\n\n for episode in range(num_episodes):\n state, _ = env.reset()\n log_probs = []\n rewards = []\n done = False\n\n while not done:\n state = torch.FloatTensor(state).unsqueeze(0)\n action_probs = policy(state)\n action_dist = torch.distributions.Categorical(action_probs)\n action = action_dist.sample()\n\n log_probs.append(action_dist.log_prob(action))\n next_state, reward, done, _, _ = env.step(action.item())\n rewards.append(reward)\n state = next_state\n\n # Compute discounted rewards\n returns = []\n G = 0\n for r in reversed(rewards):\n G = r + gamma * G\n returns.insert(0, G)\n\n returns = torch.tensor(returns)\n returns = (returns - returns.mean()) \/ (returns.std() + 1e-9) # Normalize for stability\n\n # Compute policy gradient loss\n loss = []\n for log_prob, G in zip(log_probs, returns):\n loss.append(-log_prob * G) # Gradient ascent on expected return\n loss = torch.stack(loss).sum()\n\n # Optimize policy\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n reward_history.append(sum(rewards))\n\n return reward_history<\/code><\/pre>\n\u00a0 \ud83d\udd17 Full implementation available here\u00a0<\/a><\/p>\nCode Explanation<\/h3>\n
The train_policy_gradient <\/i><\/b>function implements the REINFORCE <\/i><\/b>algorithm, which optimizes policy parameters using Monte Carlo updates. The training begins by initializing the environment and iterating over multiple episodes, collecting state-action-reward trajectories. For each step in an episode, an action is sampled from the policy, executed in the environment, and its corresponding reward is stored. After completing an episode, the discounted rewards are computed using the compute_discounted_rewards <\/i><\/b>function, ensuring that future rewards contribute appropriately to policy updates. These rewards are then normalized to reduce variance, making training more stable. The policy loss is calculated by multiplying the log probabilities of actions by their respective discounted rewards. Finally, the policy is updated using gradient descent, which maximizes the expected return by reinforcing actions that led to higher rewards.<\/p>\nExpected Outcomes & Justification<\/h4>\n
The training plot demonstrates how the total episode rewards evolve over 500 episodes. Initially, the agent performs poorly, as seen in the low reward values in early episodes (e.g., Episode 50: 20.0). However, as training progresses, the agent learns more effective strategies, leading to higher rewards (Episode 100: 134.0, Episode 150: 229.0). The performance peaks when the agent successfully balances the pole for the maximum time, reaching 500 rewards per episode (Episode 200, 350, and 450). However, instability is evident, as seen in the sharp reward drop in Episode 250 (26.0) and Episode 500 (9.0). This behaviour arises due to the high variance of PG methods, where updates can occasionally lead to suboptimal policies before stabilizing.<\/p>\n![\"Policy](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PG_OP1.webp\")
<\/figure>\n![\"Policy](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PG_OpP2.webp\")
<\/figure>\nThe overall trend shows increasing average rewards, indicating that the policy is improving. However, fluctuations in rewards highlight the limitation of vanilla PG methods, which motivates the need for more stable techniques like TRPO and PPO.<\/p>\n
Trust Region Policy Optimization (TRPO)\u00a0<\/h2>\n
While Policy Gradient (PG) methods like REINFORCE are effective, they suffer from high variance and instability in updates. One bad update can drastically collapse the learned policy. TRPO (Trust Region Policy Optimization) improves upon PG by ensuring updates are constrained within a trust region, preventing abrupt changes that could harm performance.<\/p>\n
Instead of using vanilla gradient descent, TRPO solves a constrained optimization problem:<\/p>\n![\"Trust](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_Eq1.webp\")
<\/figure>\nThis KL-divergence constraint ensures that the new policy is not too far from the previous policy, leading to more stable updates.<\/p>\n
TRPO Algorithm & Key Mathematical Concepts<\/h2>\n
TRPO optimizes the policy using Generalized Advantage Estimation (GAE) and Conjugate Gradient Descent.<\/p>\n
1. Generalized Advantage Estimation (GAE):<\/b> Computes an advantage function to estimate how much better an action is compared to the expected return.<\/p>\n![\"Generalized](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_Eq2.webp\")
<\/figure>\n\u00a0 where \u03b4_t<\/i><\/b>\u00a0is the TD error:\u00a0\u00a0<\/p>\n![\"Generalized](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_Eq3.webp\")
<\/figure>\n2.\u00a0Trust Region Constraint:<\/b> Ensures updates stay within a safe region using KL-divergence.<\/p>\n
\u00a0 where \u03b4\u00a0\u00a0<\/i><\/b>is the maximum step size.\u00a0\u00a0<\/p>\n![\"Trust](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_Eq4.webp\")
<\/figure>\n3.\u00a0 Conjugate Gradient Optimization:<\/b> Instead of directly computing the inverse Hessian, TRPO uses a conjugate gradient to find the optimal update direction efficiently.<\/p>\nCode Example: TRPO Training Loop<\/h2>\n
Below is the main TRPO training function, where we apply trust region updates and compute the discounted rewards and advantages. (Only the key function is shown; the full notebook link<\/a>.)<\/p>\ndef train_trpo(env, policy, num_episodes=500, gamma=0.99):\n reward_history = []\n\n for episode in range(num_episodes):\n state = env.reset()\n if isinstance(state, tuple):\n state = state[0] # Handle Gym versions that return (state, info)\n\n log_probs = []\n states = []\n actions = []\n rewards = []\n\n done = False\n while not done:\n state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n probs = policy(state_tensor)\n action_dist = torch.distributions.Categorical(probs)\n action = action_dist.sample()\n\n step_result = env.step(action.item())\n\n if len(step_result) == 5:\n next_state, reward, terminated, truncated, _ = step_result\n done = terminated or truncated # New Gym API\n else:\n next_state, reward, done, _ = step_result # Old Gym API\n\n log_probs.append(action_dist.log_prob(action))\n states.append(state_tensor)\n actions.append(action)\n rewards.append(reward)\n\n state = next_state\n\n # Compute discounted rewards and advantages\n discounted_rewards = compute_discounted_rewards(rewards, gamma)\n discounted_rewards = (discounted_rewards - discounted_rewards.mean()) \n \/ (discounted_rewards.std() + 1e-9)\n\n # Convert lists to tensors\n states = torch.cat(states)\n actions = torch.tensor(actions)\n advantages = discounted_rewards\n\n # Copy old policy before updating\n old_policy = PolicyNetwork(env.observation_space.shape[0], \n env.action_space.n)\n old_policy.load_state_dict(policy.state_dict())\n\n # Apply TRPO update\n trpo_step(policy, old_policy, states, actions, advantages)\n\n total_episode_reward = sum(rewards)\n reward_history.append(total_episode_reward)\n\n if (episode + 1) % 50 == 0:\n print(f\"Episode {episode+1}, Total Reward: {total_episode_reward}\")\n\n return reward_history<\/code><\/pre>\n\u00a0 \ud83d\udd17\u00a0Full implementation available\u00a0here\u00a0<\/a><\/p>\nCode Explanation<\/h3>\n
The train_trpo <\/i><\/b>function implements the Trust Region Policy Optimization update. The training loop initializes the environment and runs 500 episodes, collecting states, actions, and rewards for each step. The key difference from Policy Gradient (PG) is that TRPO maintains an old policy copy and updates the new policy while ensuring the update remains within a KL-divergence bound.<\/p>\n
The advantages are computed using discounted rewards and normalized to reduce variance. Finally, conjugate gradient descent is used to determine the optimal policy step direction. Unlike standard gradient updates, TRPO restricts step size to prevent drastic policy changes, leading to more stable performance.<\/p>\n
Expected Outcomes & Justification<\/h3>\n
The training curve for TRPO exhibits significant reward fluctuations, and the numerical results indicate that the policy does not consistently improve over time as shown below.<\/p>\n![\"Expected](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_OP1.webp\")
<\/figure>\n![\"Expected](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/TRPO_OpP2.webp\")
<\/figure>\nUnlike Policy Gradient (PG), which showed steady learning progress, TRPO struggles to maintain consistent improvements. Despite its theoretical advantages (trust region constraint preventing catastrophic updates), the actual results show high instability. The total rewards oscillate between low values (9-20), indicating that the agent fails to learn an optimal strategy efficiently.<\/p>\n
This is a known issue with TRPO\u2014it requires careful tuning of KL divergence constraints, and in many cases, the update process is computationally expensive and prone to suboptimal convergence. The reward fluctuations suggest that the agent isn\u2019t exploiting learned knowledge effectively, reinforcing the need for a more practical and robust policy optimization method.\u00a0PPO simplifies TRPO by approximating the trust region constraint using a clipped objective function, leading to faster and more efficient training.\u00a0<\/p>\n
Proximal Policy Optimization (PPO)<\/h2>\n
TRPO ensures stable policy updates but is computationally expensive due to solving a constrained optimization problem at each step. PPO (Proximal Policy Optimization) simplifies this process by using a clipped objective function to restrict updates without requiring second-order optimization.<\/p>\n
Instead of solving:<\/p>\n![\"Proximal](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_Eq1.webp\")
<\/figure>\nPPO modifies the objective function by introducing a clipped surrogate loss:<\/p>\n![\"Proximal](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_Eq2.webp\")
<\/figure>\nwhere:<\/p>\n
\n- r_t\u200b(\u03b8<\/i><\/b>)<\/i><\/b> is the probability ratio between new and old policies.<\/li>\n
- A_<\/i><\/b>t\u200b<\/i><\/b> is the advantage estimate.<\/li>\n
- \u03f5<\/i><\/b> is a small constant (e.g., 0.2) that limits excessive policy updates.<\/li>\n<\/ul>\n
This prevents overshooting updates, making PPO more computationally efficient while retaining TRPO\u2019s stability.<\/p>\n
PPO Algorithm & Key Mathematical Concept<\/h2>\n
1. Advantage Estimation using GAE:<\/b> PPO improves TRPO by using Generalized Advantage Estimation (GAE) to compute stable gradients:\u00a0\u00a0<\/p>\n![\"PPO](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_Eq3.webp\")
<\/figure>\n\u00a0 where \u03b4_t <\/i><\/b>= r_t\u00a0<\/i><\/b>+\u00a0\u03b3V(s_(t+1))\u00a0<\/i><\/b>\u2212V(s_t)<\/i><\/b>.\u00a0\u00a0<\/p>\n
2.\u00a0<\/b>Clipped Objective Function:<\/strong> Unlike TRPO, which enforces a strict KL constraint, PPO approximates the constraint using clipping:<\/p>\n![\"PPO](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_Eq4.webp\")
<\/figure>\nThis ensures that the update does not move too far, preventing policy collapse.<\/p>\n
3.\u00a0<\/b>Mini-Batch Training:<\/strong> Instead of updating the policy after each episode, PPO trains using mini-batches over multiple epochs, improving sample efficiency.<\/p>\nCode Example: PPO Training Loop<\/h2>\n
Below is the main PPO training function, where we compute advantages, apply clipped policy updates, and use mini-batches for stable learning. (Only the key function is shown; full notebook link<\/a>.)<\/p>\ndef train_ppo(env, policy, optimizer, num_episodes=500, gamma=0.99, lambda_=0.95, epsilon=0.2, batch_size=32, epochs=5):\n reward_history = []\n\n for episode in range(num_episodes):\n state = env.reset()\n if isinstance(state, tuple):\n state = state[0] # Handle Gym versions returning (state, info)\n\n log_probs = []\n values = []\n states = []\n actions = []\n rewards = []\n\n done = False\n while not done:\n state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n probs = policy(state_tensor)\n action_dist = torch.distributions.Categorical(probs)\n action = action_dist.sample()\n\n step_result = env.step(action.item())\n\n # Handle different Gym API versions\n if len(step_result) == 5:\n next_state, reward, terminated, truncated, _ = step_result\n done = terminated or truncated # New API\n else:\n next_state, reward, done, _ = step_result # Old API\n\n log_probs.append(action_dist.log_prob(action))\n states.append(state_tensor)\n actions.append(action)\n rewards.append(reward)\n\n state = next_state\n\n # Compute advantages\n values = [0] * len(rewards) # Placeholder for value estimates (since we use policy-only PPO)\n advantages = compute_advantages(rewards, values, gamma, lambda_)\n advantages = (advantages - advantages.mean()) \/ (advantages.std() + 1e-9) # Normalize advantages\n\n # Convert lists to tensors\n states = torch.cat(states)\n actions = torch.tensor(actions)\n old_log_probs = torch.tensor(log_probs)\n\n # PPO Training Loop\n for _ in range(epochs):\n for i in range(0, len(states), batch_size):\n batch_indices = slice(i, i + batch_size)\n\n new_probs = policy(states[batch_indices])\n new_action_dist = torch.distributions.Categorical(new_probs)\n new_log_probs = new_action_dist.log_prob(actions[batch_indices])\n\n loss = ppo_loss(old_log_probs[batch_indices], new_log_probs, advantages[batch_indices], epsilon)\n\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n total_episode_reward = sum(rewards)\n reward_history.append(total_episode_reward)\n\n if (episode + 1) % 50 == 0:\n print(f\"Episode {episode+1}, Total Reward: {total_episode_reward}\")\n\n return reward_history<\/code><\/pre>\n\ud83d\udd17\u00a0Full implementation available here\u00a0<\/a><\/p>\nCode Explanation<\/h3>\n
The train_ppo <\/i><\/b>function implements Proximal Policy Optimization (PPO) using a clipped surrogate loss and mini-batch updates. Unlike TRPO, which computes trust region constraints, PPO approximates them by clipping policy updates, making it much more efficient.<\/p>\n\n- The function begins by collecting episode trajectories (states, actions, log probabilities, and rewards).<\/li>\n
- Advantage estimation is computed using Generalized Advantage Estimation (GAE).<\/li>\n
- Mini-batches are used to update the policy over multiple epochs, improving sample efficiency.<\/li>\n
- Instead of a strict KL divergence constraint, PPO applies a clipped loss function to prevent destructive updates.<\/li>\n<\/ul>\n
Expected Outcomes for PPO<\/h3>\n
The PPO training curve and numerical results show a clear improvement in policy learning over time:<\/p>\n![\"PPO](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_OP1.webp\")
<\/figure>\n![\"PPO](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/PPO_OP2.webp\")
<\/figure>\nKey Observations:<\/h4>\n\n- Stable Improvement: <\/b>The early rewards (Ep 50-100) are low, indicating the agent is still exploring.<\/li>\n
- Steady Progress: <\/b>By Episode 200, the total reward surpasses 200, showing the agent is learning a structured policy.<\/li>\n
- Fluctuations Exist, But Recovery is Fast:<\/b> Between Ep 300-400, rewards drop, but PPO stabilizes and quickly rebounds to peak performance (500).<\/li>\n
- \u00a0Final Convergence:<\/b> The model reaches 500 rewards (max score) by Ep 500, confirming PPO effectively learns an optimal strategy.<\/li>\n<\/ul>\n
Compared to TRPO, PPO exhibits:<\/strong><\/p>\n\n- Less noisy training<\/li>\n
- Faster convergence<\/li>\n
- More efficient sample utilization: These improvements validate PPO\u2019s clipped updates and mini-batch training as a superior approach to policy learning.<\/li>\n<\/ul>\n
PPO is excellent for reward-based learning, but it struggles with preference-based fine-tuning in applications like LLMs (e.g., ChatGPT, DeepSeek, Claude, Gemini). DPO (Direct Preference Optimization) improves upon PPO by directly learning from human preference data instead of optimizing pure rewards.<\/p>\n
Direct Preference Optimization (DPO) \u2013 Preference Learning for LLMs<\/h2>\n
Traditional reinforcement learning (RL) techniques are designed to optimize numerical reward-based objectives. However, Large Language Models (LLMs) like ChatGPT, DeepSeek, Claude, and Gemini require fine-tuning that aligns with human preferences rather than just maximizing a reward function. This is where Direct Preference Optimization (DPO) plays a crucial role. Unlike RL-based methods like PPO, which rely on an explicitly trained reward model, DPO optimizes models directly using human feedback. By leveraging preference pairs (where one response is preferred over another), DPO enables models to learn human-like responses efficiently.<\/p>\n
DPO eliminates the need for a separate reward model, making it a simpler and more data-driven approach compared to Reinforcement Learning from Human Feedback (RLHF). Instead of reward-based fine-tuning, DPO updates the model parameters to increase the probability of preferred responses while decreasing the probability of rejected responses. This makes the training process more stable and avoids the complexities of RL algorithms like PPO, which involve constrained policy updates and KL penalties.<\/p>\n
The significance of DPO lies in its ability to fine-tune LLMs in a way that ensures better response alignment with human expectations. By removing explicit reward models, it prevents the instability often associated with RL-based fine-tuning. Moreover, DPO reduces the risk of harmful, misleading, or biased outputs, making LLMs safer and more reliable. This streamlined optimization process makes it a practical alternative to RL-based fine-tuning, especially when human preference data is available at scale.<\/p>\n
The DPO Training Dataset<\/h3>\n
For DPO, we use human preference data, where each prompt has a preferred response and a rejected response.<\/p>\n
Example Preference Dataset (Used for Fine-Tuning)<\/p>\n
preference_data = [\n {\"prompt\": \"What is the capital of France?\",\n \"preferred\": \"The capital of France is Paris.\",\n \"rejected\": \"France is a country in Europe.\"},\n\n {\"prompt\": \"Who wrote Hamlet?\",\n \"preferred\": \"Hamlet was written by William Shakespeare.\",\n \"rejected\": \"Hamlet is an old book.\"},\n\n {\"prompt\": \"Tell me a joke.\",\n \"preferred\": \"Why did the scarecrow win an award? Because he was outstanding in his field!\",\n \"rejected\": \"I don\u2019t know any jokes.\"},\n\n {\"prompt\": \"What is artificial intelligence?\",\n \"preferred\": \"Artificial intelligence is the simulation of human intelligence in machines.\",\n \"rejected\": \"AI is just robots.\"},\n\n {\"prompt\": \"How to stay motivated?\",\n \"preferred\": \"Set clear goals, track progress, and reward yourself for achievements.\",\n \"rejected\": \"Just be motivated.\"},\n]\n<\/code><\/pre>\nThe preferred responses are accurate, informative, and well-structured, while the rejected responses are vague, incorrect, or unhelpful.<\/p>\n
The DPO Loss Function<\/h3>\n
DPO is formulated as a pairwise ranking problem between a preferred response and a rejected response for the same prompt. The goal is to increase the log probability of preferred responses while decreasing the probability of rejected ones.<\/p>\n
Mathematically, the DPO objective is:<\/p>\n![\"The](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/DPO_Eq1.webp\")
<\/figure>\nWhere:<\/p>\n
\n- y^+<\/b> is the preferred response<\/li>\n
- y^- <\/i><\/b>is the rejected response<\/li>\n
- \u03b2<\/b><\/i> is a scaling hyperparameter controlling preference strength<\/li>\n
- P_\u03b8(y\u2223x)<\/i><\/b> is the log probability of generating a response given input x<\/i><\/b><\/li>\n<\/ul>\n
This is similar to logistic regression, where the model maximizes separation between preferred and rejected responses.<\/p>\n
Code Example: Direct Preference Optimization (DPO)<\/h2>\n
DPO fine-tunes LLMs by training on human-labeled preference pairs. The core logic of DPO training involves optimizing model weights based on preferred vs. rejected responses. The function below trains a transformer-based model to increase the likelihood of preferred responses while decreasing the likelihood of rejected ones. Below is the key function for computing the DPO loss and updating the model (only the main function is shown for scope; full notebook is linked<\/a>).<\/p>\ndef dpo_loss(preferred_log_probs, rejected_log_probs, beta=0.1):\n \"\"\"Computes the DPO loss function to optimize based on preferences\"\"\"\n return -torch.mean(torch.sigmoid(beta * (preferred_log_probs - \n rejected_log_probs)))\n\ndef encode_text(prompt, response):\n \"\"\"Encodes the prompt + response into tokenized format with proper padding\"\"\"\n tokenizer.pad_token = tokenizer.eos_token # Fix padding issue\n input_text = f\"User: {prompt}\\nAssistant: {response}\"\n\n inputs = tokenizer(\n input_text,\n return_tensors=\"pt\",\n padding=True, # Enable padding\n truncation=True, # Truncate if too long\n max_length=512 # Set max length for safety\n )\n\n return inputs[\"input_ids\"], inputs[\"attention_mask\"]\n\nloss_history = [] # Store loss values\n\noptimizer = optim.AdamW(model.parameters(), lr=5e-5)\n\nfor epoch in range(10): # Train for 10 epochs\n total_loss = 0\n\n for data in preference_data:\n prompt, preferred, rejected = data[\"prompt\"], data[\"preferred\"], \n data[\"rejected\"]\n\n # Encode preferred and rejected responses\n pref_input_ids, pref_attention_mask = encode_text(prompt, preferred)\n rej_input_ids, rej_attention_mask = encode_text(prompt, rejected)\n\n # Get log probabilities from the model\n preferred_logits = model(pref_input_ids, attention_mask=\n pref_attention_mask).logits[:, -1, :]\n rejected_logits = model(rej_input_ids, attention_mask=rej_attention_mask)\n .logits[:, -1, :]\n\n preferred_log_probs = preferred_logits.log_softmax(dim=-1)\n rejected_log_probs = rejected_logits.log_softmax(dim=-1)\n\n # Compute DPO loss\n loss = dpo_loss(preferred_log_probs, rejected_log_probs, beta=0.5)\n\n # Optimize the model\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n total_loss += loss.item()\n\n loss_history.append(total_loss) # Store loss for visualization\n print(f\"Epoch {epoch + 1}, Loss: {total_loss:.4f}\")\n<\/code><\/pre>\n\ud83d\udd17\u00a0Full implementation available here\u00a0<\/a><\/p>\nExpected Output & Analysis<\/h3>\n
The outcomes of Direct Preference Optimization (DPO) can be analyzed from multiple angles: loss convergence, probability shifts, and qualitative response improvements. The training loss curve shows a sharp drop in the initial epochs, followed by stabilization, indicating that the model quickly learns to align with human preferences. The plateau in loss suggests that further optimization yields diminishing improvements, confirming effective preference-based fine-tuning.<\/p>\n![\"Direct](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/DPO_OP1.webp\")
<\/figure>\n![\"Direct](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/DPO_OP3.webp\")
<\/figure>\nThe probability shift visualization reveals that preferred responses consistently achieve higher log probabilities than rejected ones. This confirms that DPO successfully adjusts the model\u2019s behaviour, reinforcing the correct responses while suppressing undesired ones. Some variance in probability shifts suggests that certain prompts may still require fine-tuning for optimal alignment.<\/p>\n![\"DPO](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/DPO_OP2.webp\")
<\/figure>\nA direct comparison of model responses before and after DPO fine-tuning highlights clear improvements. Initially, the model fails to generate a joke, instead providing an irrelevant response. After fine-tuning, it attempts humor but still lacks coherence. This demonstrates that while DPO enhances preference alignment, additional refinements or complementary techniques may be required to generate high-quality, structured responses.<\/p>\n![\"DPO\"](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/DPO_OP4.webp\")
<\/figure>\nAlthough DPO effectively tunes LLMs without an explicit reward function, it lacks the structured policy learning of reinforcement learning-based methods. This is where General Reinforcement Pretraining Optimization (GRPO) by DeepSeek comes in, combining the strengths of DPO and PPO to enhance LLM fine-tuning further. The next section will explore how GRPO refines policy optimization for large-scale models.<\/p>\n
GRPO \u2013 Group Relative Policy Optimization (DeepSeek\u2019s Approach)<\/h2>\n
DeepSeek\u2019s Group Relative Policy Optimization (GRPO) is an advanced preference optimization technique that extends Direct Preference Optimization (DPO) while incorporating elements from Proximal Policy Optimization (PPO). Unlike traditional policy optimization methods that operate on single preference pairs, GRPO leverages group-wise preference ranking, enabling better alignment with human feedback in large-scale LLM fine-tuning.<\/p>\n
Traditional preference-based optimization methods, such as DPO (Direct Preference Optimization), operate on pairwise comparisons\u2014one preferred and one rejected response. However, this approach fails to scale efficiently when optimizing on large datasets where multiple responses per prompt are ranked in order of preference. To address this limitation, DeepSeek introduced Group Relative Policy Optimization (GRPO), which allows group-based preference ranking rather than just single-pair preference updates. Instead of comparing two responses at a time, GRPO compares all ranked responses within a batch and optimizes the policy accordingly.<\/p>\n
Mathematically, GRPO extends DPO\u2019s reward-free optimization by defining an ordered preference ranking among multiple completions and optimizing their relative likelihoods accordingly.<\/p>\n
Mathematical Foundation of GRPO<\/h2>\n
Since this is the main intent behind the blog, we will dive deep into the mathematics of this.<\/p>\n
1. Expected Return in Preference Optimization<\/h3>\n
In standard reinforcement learning, the expected return of a policy \u03c0_\u03b8\u00a0<\/i><\/b>is:<\/p>\n![\"Expected](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_Eq1.webp\")
<\/figure>\nwhere R(s_t,a_t)<\/i><\/b>\u00a0is the reward at timestep t<\/i><\/b>.<\/p>\n
However, LLM fine-tuning does not operate in traditional reward-based RL. Instead, we optimize over human preferences, meaning that reward models are unnecessary.<\/p>\n
Instead of learning a reward function, GRPO directly optimizes the model parameters to increase the likelihood of higher-ranked responses over lower-ranked ones.<\/p>\n
2. Ranking-Based Probability Optimization<\/h3>\n
Given a set of responses r_1,r_2, \u2026, r_n<\/i><\/b>\u00a0ranked in order of preference, we define a likelihood ratio:<\/p>\n![\"Ranking-Based](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_Eq2.webp\")
<\/figure>\nwhere x\u00a0<\/i><\/b>is the input prompt, and \u03c0_\u03b8\u00a0<\/i><\/b>represents the policy (LLM) parameterized by \u03b8<\/b><\/i>. The key objective is to maximize the probability of higher-ranked responses while suppressing the probability of lower-ranked ones.<\/p>\n
To enforce relative preference constraints, GRPO optimizes the following pairwise ranking loss across all response pairs:<\/p>\n![\"Ranking-Based](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_Eq3.webp\")
<\/figure>\nwhere:<\/p>\n
\n- \u03c3(x)<\/i><\/b> is the sigmoid function ensuring probability normalization<\/li>\n
- \u03b2<\/i><\/b> is a temperature scaling parameter controlling gradient magnitude.<\/li>\n
- \u03c0_<\/i><\/b>\u03b8\u200b<\/i><\/b> is the policy (LLM).<\/li>\n
- The sum iterates over all pairs (i, j) where r_i<\/i><\/b>, is ranked higher than r_j<\/i><\/b>.<\/li>\n<\/ul>\n
The KL-regularized version of GRPO adds a penalty term to prevent drastic shifts in model behaviour:<\/p>\n![\"KL-regularized](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_Eq4.webp\")
<\/figure>\nwhere D_KL<\/i><\/b>\u200b ensures conservative updates to prevent overfitting.<\/p>\nData for GRPO Fine-Tuning<\/h2>\n
Below is an example dataset used to fine-tune an LLM using ranked preferences:<\/p>\n
grpo_preference_data = [\n {\"prompt\": \"What is the capital of France?\",\n \"responses\": [\n {\"text\": \"The capital of France is Paris.\", \"rank\": 1},\n {\"text\": \"Paris is the largest city in France.\", \"rank\": 2},\n {\"text\": \"Paris is in France.\", \"rank\": 3},\n {\"text\": \"France is a country in Europe.\", \"rank\": 4}\n ]},\n\n {\"prompt\": \"Tell me a joke.\",\n \"responses\": [\n {\"text\": \"Why did the scarecrow win an award? Because he was outstanding \n in his field!\", \"rank\": 1},\n {\"text\": \"Why did the chicken cross the road? To get to the other side.\",\n \"rank\": 2},\n {\"text\": \"Jokes are funny.\", \"rank\": 3},\n {\"text\": \"I don\u2019t know any jokes.\", \"rank\": 4}\n ]}\n]<\/code><\/pre>\nEach prompt has multiple responses with assigned ranks. The model learns to increase the probability of higher-ranked responses while reducing the probability of lower-ranked ones.<\/p>\n
Code Implementation: Group-Based Preference Optimization<\/h2>\n
Below is the key function for computing the DPO loss and updating the model (only the main function is shown for scope; the full notebook is\u00a0linked<\/a>). The GRPO training function processes multiple ranked responses per prompt, optimizing log-likelihood differences while enforcing KL constraints.<\/p>\ndef deepseek_grpo_loss(log_probs, rankings, input_ids, beta=1.0, kl_penalty=0.02, epsilon=1e-6):\n \"\"\"Computes DeepSeek GRPO loss with pairwise ranking and KL regularization.\"\"\"\n loss_terms = []\n num_pairs = 0\n\n log_probs = torch.clamp(log_probs, min=-10, max=10) # Prevent extreme values\n\n for i in range(len(rankings)):\n for j in range(i + 1, len(rankings)):\n if rankings[i] 0 else torch.tensor(0.0, device=log_probs.device)\n\n # KL regularization to prevent policy divergence\n old_logits = base_model(input_ids).logits[:, -1, :]\n old_log_probs = old_logits.log_softmax(dim=-1)\n\n kl_div = torch.nn.functional.kl_div(log_probs, old_log_probs.clamp(min=epsilon), reduction=\"batchmean\")\n\n return loss + (kl_penalty * kl_div.mean()) # Ensure single scalar<\/code><\/pre>\nTraining Loop for GRPO<\/h2>\n
The training loop processes ranked responses, computes loss, and updates the model while enforcing stability constraints.<\/p>\n
loss_history = []\nnum_epochs = 15\n\nfor epoch in range(num_epochs):\n total_loss = 0\n\n for data in grpo_preference_data:\n prompt, responses = data[\"prompt\"], data[\"responses\"]\n\n input_ids, rankings = encode_text(prompt, responses)\n\n logits = model(input_ids).logits[:, -1, :]\n log_probs = logits.log_softmax(dim=-1)\n\n loss = deepseek_grpo_loss(log_probs, rankings, input_ids)\n\n if torch.isnan(loss):\n print(f\"Skipping update at epoch {epoch} due to NaN loss.\")\n continue\n\n optimizer.zero_grad()\n loss.backward()\n\n torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n\n optimizer.step()\n total_loss += loss.item()\n\n loss_history.append(total_loss)\n scheduler.step()\n print(f\"Epoch {epoch + 1}, Loss: {total_loss:.4f}\")\n<\/code><\/pre>\n\ud83d\udd17 Full implementation available here\u00a0<\/a><\/p>\nExpected Outcome and Results<\/h2>\n
The expected outcomes of GRPO fine-tuning on the LLM, based on the provided outputs, highlight improvements in model optimization and preference-based ranking.<\/p>\n
The training loss curve shows a gradual and stable decline over 15 epochs, indicating that the model is learning effectively. Unlike conventional policy optimization methods, GRPO ensures that ranked responses improve without drastic fluctuations, suggesting smooth convergence.<\/p>\n![\"DeepSeek](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_OP1.webp\")
<\/figure>\n![\"DeepSeek](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_OP4.webp\")
<\/figure>\nThe loss value distribution over epochs presents a histogram where most values concentrate around a decreasing trend, showing that GRPO efficiently optimizes the model while maintaining stable loss updates. This distribution further indicates that loss values do not exhibit large variations, preventing instability in preference ranking.<\/p>\n![\"distribution](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_OP2.webp\")
<\/figure>\nThe log probability distribution before vs. after fine-tuning provides crucial insights into the model\u2019s response generation. The shift in probability distribution suggests that after fine-tuning, the model assigns higher confidence to preferred responses. This shift results in responses that align better with human expectations and rankings.<\/p>\n![\"Log](\"https:\/\/cdn.analyticsvidhya.com\/wp-content\/uploads\/2025\/02\/GRPO_OP3.webp\")
<\/figure>\nOverall, the expected outcome of GRPO fine-tuning is a well-optimized model capable of generating high-quality responses ranked effectively based on preference learning. This demonstrates why GRPO is an effective alternative to traditional RL methods like PPO or DPO, offering a structured approach to optimizing LLMs without explicit reward models.<\/p>\n
Final Model Insights: Why GRPO Excels in LLM Fine-Tuning<\/h2>\n
Unlike pairwise DPO and trust-region PPO, GRPO allows LLMs to learn from multiple ranked completions per prompt, significantly improving response quality, stability, and human alignment.<\/p>\n
\n- More scalable than pairwise methods \u2192 Learns from multiple ranked completions rather than just binary comparisons.<\/li>\n
- No explicit reward modeling \u2192 Unlike RLHF, GRPO fine-tunes without requiring a trained reward model.<\/li>\n
- KL regularization stabilizes updates \u2192 Prevents catastrophic shifts in response distribution.<\/li>\n
- Better generalization across prompts \u2192 Ensures the LLM produces high-quality, human-aligned responses.<\/li>\n<\/ul>\n
With reinforcement learning playing an increasingly central role in fine-tuning LLMs, GRPO stands out as the next step in AI preference learning, setting a new standard for human-aligned language modeling.<\/p>\n
Conclusion<\/h2>\n
Policy optimization techniques play a critical role in reinforcement learning and LLM fine-tuning. Each method\u2014Policy Gradient (PG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO)\u2014offers unique advantages and trade-offs. PG serves as the foundation but suffers from high variance, while TRPO provides stability at the cost of computational complexity. PPO, being a refined version of TRPO, balances efficiency and robustness, making it widely used in RL applications. DPO, on the other hand, optimizes LLMs directly using preference data, eliminating the need for a reward model. Finally, GRPO, as introduced by DeepSeek, enhances preference-based fine-tuning by leveraging relative ranking in a structured manner.<\/p>\n
Below is a comparison of these LLM Optimization methods based on key aspects such as variance, stability, sample efficiency, and suitability for reinforcement learning versus LLM fine-tuning:<\/p>\n
\n\nMethod<\/th>\n Variance<\/th>\n Stability<\/th>\n Sample Efficiency<\/th>\n Best for<\/th>\n Limitations<\/th>\n<\/tr>\n \nPG (REINFORCE)<\/td>\n High<\/td>\n Low<\/td>\n Inefficient<\/td>\n Simple RL problems<\/td>\n High variance, slow convergence<\/td>\n<\/tr>\n \nTRPO<\/td>\n Low<\/td>\n High<\/td>\n Moderate<\/td>\n High-stability RL tasks<\/td>\n Complex second-order updates, expensive<\/td>\n<\/tr>\n \nPPO<\/td>\n Medium<\/td>\n High<\/td>\n Efficient<\/td>\n General RL tasks, Robotics, Games<\/td>\n May require careful hyperparameter tuning<\/td>\n<\/tr>\n \nDPO<\/td>\n Low<\/td>\n High<\/td>\n High<\/td>\n LLM fine-tuning with human preferences<\/td>\n Lacks explicit reinforcement learning framework<\/td>\n<\/tr>\n \nGRPO<\/td>\n Low<\/td>\n High<\/td>\n High<\/td>\n Preference-based LLM fine-tuning<\/td>\n Newer method, requires further empirical validation<\/td>\n<\/tr>\n<\/table>\nFor practitioners, the choice depends on the task at hand. If optimizing reinforcement learning agents in games or robotics, PPO is the best choice due to its balance of efficiency and performance. If high-stability optimization is required, TRPO is preferred despite its computational cost. DPO and GRPO, however, are better suited for LLM fine-tuning, with GRPO providing an even stronger optimization framework based on relative preference ranking rather than just binary preference signals.<\/p>\n
Key Takeaways<\/h3>\n
Reinforcement learning (RL) plays a crucial role in both game-playing agents and LLM fine-tuning, but the optimization techniques vary significantly.<\/p>\n
\n- PG, TRPO, and PPO are fundamental in RL, with PPO being the most practical choice for its efficiency and performance balance.<\/li>\n
- DPO introduced a major shift in LLM fine-tuning by eliminating explicit reward models, making human preference alignment easier and more efficient.<\/li>\n
- GRPO, pioneered by DeepSeek, further refines LLM fine-tuning by optimizing for relative ranking rather than just binary comparisons, improving preference-based alignment.<\/li>\n
- For RL tasks, PPO remains the dominant method, while for LLM fine-tuning, DPO and GRPO are superior choices due to their ability to fine-tune models using direct preference data without RL instability.<\/li>\n<\/ul>\n
This blog highlights how reinforcement learning and preference-based fine-tuning are converging, with new techniques like GRPO bridging the gap between structured optimization and real-world deployment of large-scale AI systems.<\/p>\n
The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion.<\/strong><\/p>\nFrequently Asked Questions<\/h2>\n\nQ1. What is the difference between PPO and DPO?<\/strong> <\/p>\nAns. PPO (Proximal Policy Optimization) is an RL-based optimization method that improves policies while maintaining stability using a clipping mechanism. It is widely used in reinforcement learning tasks such as robotics and game-playing AI. DPO (Direct Preference Optimization), on the other hand, is designed specifically for LLM fine-tuning, directly optimizing the model based on human preferences without requiring an explicit reward model. DPO is simpler and more efficient for aligning language models with human intent.<\/p>\n<\/p><\/div>\n
This article was published as a part of the\u00a0<\/strong><\/em>Data Science Blogathon.<\/strong><\/em><\/a><\/p>\n Before diving into DeepSeek\u2019s GRPO, it\u2019s crucial to understand the policy optimization techniques that form the foundation of reinforcement learning (RL) in both traditional control tasks and LLM fine-tuning<\/a>. Policy optimization refers to the process of improving an AI agent\u2019s decision-making strategy (policy) to maximize expected rewards. While early methods like vanilla policy gradient (PG)<\/a> laid the groundwork, more sophisticated techniques such as TRPO, PPO, DPO, and GRPO evolved to address issues like stability, efficiency, and preference alignment.<\/p>\n \u00a0 At its core, policy optimization is about learning the optimal policy \u03c0_\u03b8(a\u2223s)<\/b>, which maps a state s<\/i>\u00a0to an action a<\/i>\u00a0while maximizing long-term rewards. The objective function in RL is typically formulated as:\u00a0\u00a0<\/p>\n Where R(\u03c4)<\/b>\u00a0is the total reward collected in a trajectory \u03c4, and the expectation is taken over all possible trajectories following policy \u03c0_\u03b8<\/b>.<\/p>\n There are three major approaches to policy optimization:<\/p>\n RL is typically formulated as a Markov Decision Process<\/b> (MDP), represented as:<\/p>\n where:<\/p>\n \u00a0 The Expected Return<\/b> (ER) measures how much cumulative reward we expect from following policy \u03c0_\u03b8<\/b>:\u00a0\u00a0<\/p>\n \u00a0 where \u03b3<\/b> (0 \u2264 \u03b3<\/b> \u2264 1) determines how much future rewards contribute.\u00a0\u00a0<\/p>\n Policy gradient (PG) methods update the policy using gradients of expected rewards. The key equation:<\/p>\n where:<\/p>\n To reduce variance in gradient estimates, we use the advantage function:<\/p>\n where:<\/p>\n Using A(s,a)<\/b>\u00a0helps make updates more stable and efficient.<\/p>\n The Policy Gradient (PG) method is the most fundamental approach to reinforcement learning. Instead of learning a value function, PG directly parameterizes the policy \u03c0_\u03b8(a\u2223s)\u00a0<\/b>and updates it using gradient ascent. This allows learning in continuous action spaces, making it effective for tasks like robotics, game AI, and LLM fine-tuning.<\/p>\n However, PG methods suffer from high variance due to their reliance on sampling full trajectories. More advanced methods like TRPO, PPO, and GRPO build upon PG to improve stability.<\/p>\n \u00a0 The goal of policy optimization is to find policy parameters \u03b8<\/b>\u00a0that maximize expected return:\u00a0\u00a0<\/p>\n Using the log-derivative trick, we obtain the Policy Gradient Theorem:<\/p>\n where:<\/p>\n The REINFORCE algorithm is the simplest form of PG. It samples trajectories, computes rewards, and updates the policy parameters. Below is the main training loop (only the key function is shown to limit the scope; the full notebook is linked<\/a>).<\/p>\n \u00a0 \ud83d\udd17 Full implementation available here\u00a0<\/a><\/p>\n The train_policy_gradient <\/i><\/b>function implements the REINFORCE <\/i><\/b>algorithm, which optimizes policy parameters using Monte Carlo updates. The training begins by initializing the environment and iterating over multiple episodes, collecting state-action-reward trajectories. For each step in an episode, an action is sampled from the policy, executed in the environment, and its corresponding reward is stored. After completing an episode, the discounted rewards are computed using the compute_discounted_rewards <\/i><\/b>function, ensuring that future rewards contribute appropriately to policy updates. These rewards are then normalized to reduce variance, making training more stable. The policy loss is calculated by multiplying the log probabilities of actions by their respective discounted rewards. Finally, the policy is updated using gradient descent, which maximizes the expected return by reinforcing actions that led to higher rewards.<\/p>\n The training plot demonstrates how the total episode rewards evolve over 500 episodes. Initially, the agent performs poorly, as seen in the low reward values in early episodes (e.g., Episode 50: 20.0). However, as training progresses, the agent learns more effective strategies, leading to higher rewards (Episode 100: 134.0, Episode 150: 229.0). The performance peaks when the agent successfully balances the pole for the maximum time, reaching 500 rewards per episode (Episode 200, 350, and 450). However, instability is evident, as seen in the sharp reward drop in Episode 250 (26.0) and Episode 500 (9.0). This behaviour arises due to the high variance of PG methods, where updates can occasionally lead to suboptimal policies before stabilizing.<\/p>\n The overall trend shows increasing average rewards, indicating that the policy is improving. However, fluctuations in rewards highlight the limitation of vanilla PG methods, which motivates the need for more stable techniques like TRPO and PPO.<\/p>\n While Policy Gradient (PG) methods like REINFORCE are effective, they suffer from high variance and instability in updates. One bad update can drastically collapse the learned policy. TRPO (Trust Region Policy Optimization) improves upon PG by ensuring updates are constrained within a trust region, preventing abrupt changes that could harm performance.<\/p>\n Instead of using vanilla gradient descent, TRPO solves a constrained optimization problem:<\/p>\n This KL-divergence constraint ensures that the new policy is not too far from the previous policy, leading to more stable updates.<\/p>\n TRPO optimizes the policy using Generalized Advantage Estimation (GAE) and Conjugate Gradient Descent.<\/p>\n 1. Generalized Advantage Estimation (GAE):<\/b> Computes an advantage function to estimate how much better an action is compared to the expected return.<\/p>\n \u00a0 where \u03b4_t<\/i><\/b>\u00a0is the TD error:\u00a0\u00a0<\/p>\n 2.\u00a0Trust Region Constraint:<\/b> Ensures updates stay within a safe region using KL-divergence.<\/p>\n \u00a0 where \u03b4\u00a0\u00a0<\/i><\/b>is the maximum step size.\u00a0\u00a0<\/p>\n 3.\u00a0 Conjugate Gradient Optimization:<\/b> Instead of directly computing the inverse Hessian, TRPO uses a conjugate gradient to find the optimal update direction efficiently.<\/p>\n Below is the main TRPO training function, where we apply trust region updates and compute the discounted rewards and advantages. (Only the key function is shown; the full notebook link<\/a>.)<\/p>\n \u00a0 \ud83d\udd17\u00a0Full implementation available\u00a0here\u00a0<\/a><\/p>\n The train_trpo <\/i><\/b>function implements the Trust Region Policy Optimization update. The training loop initializes the environment and runs 500 episodes, collecting states, actions, and rewards for each step. The key difference from Policy Gradient (PG) is that TRPO maintains an old policy copy and updates the new policy while ensuring the update remains within a KL-divergence bound.<\/p>\n The advantages are computed using discounted rewards and normalized to reduce variance. Finally, conjugate gradient descent is used to determine the optimal policy step direction. Unlike standard gradient updates, TRPO restricts step size to prevent drastic policy changes, leading to more stable performance.<\/p>\n The training curve for TRPO exhibits significant reward fluctuations, and the numerical results indicate that the policy does not consistently improve over time as shown below.<\/p>\n Unlike Policy Gradient (PG), which showed steady learning progress, TRPO struggles to maintain consistent improvements. Despite its theoretical advantages (trust region constraint preventing catastrophic updates), the actual results show high instability. The total rewards oscillate between low values (9-20), indicating that the agent fails to learn an optimal strategy efficiently.<\/p>\n This is a known issue with TRPO\u2014it requires careful tuning of KL divergence constraints, and in many cases, the update process is computationally expensive and prone to suboptimal convergence. The reward fluctuations suggest that the agent isn\u2019t exploiting learned knowledge effectively, reinforcing the need for a more practical and robust policy optimization method.\u00a0PPO simplifies TRPO by approximating the trust region constraint using a clipped objective function, leading to faster and more efficient training.\u00a0<\/p>\n TRPO ensures stable policy updates but is computationally expensive due to solving a constrained optimization problem at each step. PPO (Proximal Policy Optimization) simplifies this process by using a clipped objective function to restrict updates without requiring second-order optimization.<\/p>\n Instead of solving:<\/p>\n PPO modifies the objective function by introducing a clipped surrogate loss:<\/p>\n where:<\/p>\n This prevents overshooting updates, making PPO more computationally efficient while retaining TRPO\u2019s stability.<\/p>\n 1. Advantage Estimation using GAE:<\/b> PPO improves TRPO by using Generalized Advantage Estimation (GAE) to compute stable gradients:\u00a0\u00a0<\/p>\n \u00a0 where \u03b4_t <\/i><\/b>= r_t\u00a0<\/i><\/b>+\u00a0\u03b3V(s_(t+1))\u00a0<\/i><\/b>\u2212V(s_t)<\/i><\/b>.\u00a0\u00a0<\/p>\n 2.\u00a0<\/b>Clipped Objective Function:<\/strong> Unlike TRPO, which enforces a strict KL constraint, PPO approximates the constraint using clipping:<\/p>\n This ensures that the update does not move too far, preventing policy collapse.<\/p>\n 3.\u00a0<\/b>Mini-Batch Training:<\/strong> Instead of updating the policy after each episode, PPO trains using mini-batches over multiple epochs, improving sample efficiency.<\/p>\n Below is the main PPO training function, where we compute advantages, apply clipped policy updates, and use mini-batches for stable learning. (Only the key function is shown; full notebook link<\/a>.)<\/p>\n \ud83d\udd17\u00a0Full implementation available here\u00a0<\/a><\/p>\n The train_ppo <\/i><\/b>function implements Proximal Policy Optimization (PPO) using a clipped surrogate loss and mini-batch updates. Unlike TRPO, which computes trust region constraints, PPO approximates them by clipping policy updates, making it much more efficient.<\/p>\n The PPO training curve and numerical results show a clear improvement in policy learning over time:<\/p>\n Compared to TRPO, PPO exhibits:<\/strong><\/p>\n PPO is excellent for reward-based learning, but it struggles with preference-based fine-tuning in applications like LLMs (e.g., ChatGPT, DeepSeek, Claude, Gemini). DPO (Direct Preference Optimization) improves upon PPO by directly learning from human preference data instead of optimizing pure rewards.<\/p>\n Traditional reinforcement learning (RL) techniques are designed to optimize numerical reward-based objectives. However, Large Language Models (LLMs) like ChatGPT, DeepSeek, Claude, and Gemini require fine-tuning that aligns with human preferences rather than just maximizing a reward function. This is where Direct Preference Optimization (DPO) plays a crucial role. Unlike RL-based methods like PPO, which rely on an explicitly trained reward model, DPO optimizes models directly using human feedback. By leveraging preference pairs (where one response is preferred over another), DPO enables models to learn human-like responses efficiently.<\/p>\n DPO eliminates the need for a separate reward model, making it a simpler and more data-driven approach compared to Reinforcement Learning from Human Feedback (RLHF). Instead of reward-based fine-tuning, DPO updates the model parameters to increase the probability of preferred responses while decreasing the probability of rejected responses. This makes the training process more stable and avoids the complexities of RL algorithms like PPO, which involve constrained policy updates and KL penalties.<\/p>\n The significance of DPO lies in its ability to fine-tune LLMs in a way that ensures better response alignment with human expectations. By removing explicit reward models, it prevents the instability often associated with RL-based fine-tuning. Moreover, DPO reduces the risk of harmful, misleading, or biased outputs, making LLMs safer and more reliable. This streamlined optimization process makes it a practical alternative to RL-based fine-tuning, especially when human preference data is available at scale.<\/p>\n For DPO, we use human preference data, where each prompt has a preferred response and a rejected response.<\/p>\n Example Preference Dataset (Used for Fine-Tuning)<\/p>\n The preferred responses are accurate, informative, and well-structured, while the rejected responses are vague, incorrect, or unhelpful.<\/p>\n DPO is formulated as a pairwise ranking problem between a preferred response and a rejected response for the same prompt. The goal is to increase the log probability of preferred responses while decreasing the probability of rejected ones.<\/p>\n Mathematically, the DPO objective is:<\/p>\n Where:<\/p>\n This is similar to logistic regression, where the model maximizes separation between preferred and rejected responses.<\/p>\n DPO fine-tunes LLMs by training on human-labeled preference pairs. The core logic of DPO training involves optimizing model weights based on preferred vs. rejected responses. The function below trains a transformer-based model to increase the likelihood of preferred responses while decreasing the likelihood of rejected ones. Below is the key function for computing the DPO loss and updating the model (only the main function is shown for scope; full notebook is linked<\/a>).<\/p>\n \ud83d\udd17\u00a0Full implementation available here\u00a0<\/a><\/p>\n The outcomes of Direct Preference Optimization (DPO) can be analyzed from multiple angles: loss convergence, probability shifts, and qualitative response improvements. The training loss curve shows a sharp drop in the initial epochs, followed by stabilization, indicating that the model quickly learns to align with human preferences. The plateau in loss suggests that further optimization yields diminishing improvements, confirming effective preference-based fine-tuning.<\/p>\n The probability shift visualization reveals that preferred responses consistently achieve higher log probabilities than rejected ones. This confirms that DPO successfully adjusts the model\u2019s behaviour, reinforcing the correct responses while suppressing undesired ones. Some variance in probability shifts suggests that certain prompts may still require fine-tuning for optimal alignment.<\/p>\n A direct comparison of model responses before and after DPO fine-tuning highlights clear improvements. Initially, the model fails to generate a joke, instead providing an irrelevant response. After fine-tuning, it attempts humor but still lacks coherence. This demonstrates that while DPO enhances preference alignment, additional refinements or complementary techniques may be required to generate high-quality, structured responses.<\/p>\n Although DPO effectively tunes LLMs without an explicit reward function, it lacks the structured policy learning of reinforcement learning-based methods. This is where General Reinforcement Pretraining Optimization (GRPO) by DeepSeek comes in, combining the strengths of DPO and PPO to enhance LLM fine-tuning further. The next section will explore how GRPO refines policy optimization for large-scale models.<\/p>\n DeepSeek\u2019s Group Relative Policy Optimization (GRPO) is an advanced preference optimization technique that extends Direct Preference Optimization (DPO) while incorporating elements from Proximal Policy Optimization (PPO). Unlike traditional policy optimization methods that operate on single preference pairs, GRPO leverages group-wise preference ranking, enabling better alignment with human feedback in large-scale LLM fine-tuning.<\/p>\n Traditional preference-based optimization methods, such as DPO (Direct Preference Optimization), operate on pairwise comparisons\u2014one preferred and one rejected response. However, this approach fails to scale efficiently when optimizing on large datasets where multiple responses per prompt are ranked in order of preference. To address this limitation, DeepSeek introduced Group Relative Policy Optimization (GRPO), which allows group-based preference ranking rather than just single-pair preference updates. Instead of comparing two responses at a time, GRPO compares all ranked responses within a batch and optimizes the policy accordingly.<\/p>\n Mathematically, GRPO extends DPO\u2019s reward-free optimization by defining an ordered preference ranking among multiple completions and optimizing their relative likelihoods accordingly.<\/p>\n Since this is the main intent behind the blog, we will dive deep into the mathematics of this.<\/p>\n In standard reinforcement learning, the expected return of a policy \u03c0_\u03b8\u00a0<\/i><\/b>is:<\/p>\n where R(s_t,a_t)<\/i><\/b>\u00a0is the reward at timestep t<\/i><\/b>.<\/p>\n However, LLM fine-tuning does not operate in traditional reward-based RL. Instead, we optimize over human preferences, meaning that reward models are unnecessary.<\/p>\n Instead of learning a reward function, GRPO directly optimizes the model parameters to increase the likelihood of higher-ranked responses over lower-ranked ones.<\/p>\n Given a set of responses r_1,r_2, \u2026, r_n<\/i><\/b>\u00a0ranked in order of preference, we define a likelihood ratio:<\/p>\n where x\u00a0<\/i><\/b>is the input prompt, and \u03c0_\u03b8\u00a0<\/i><\/b>represents the policy (LLM) parameterized by \u03b8<\/b><\/i>. The key objective is to maximize the probability of higher-ranked responses while suppressing the probability of lower-ranked ones.<\/p>\n To enforce relative preference constraints, GRPO optimizes the following pairwise ranking loss across all response pairs:<\/p>\n where:<\/p>\n The KL-regularized version of GRPO adds a penalty term to prevent drastic shifts in model behaviour:<\/p>\n where D_KL<\/i><\/b>\u200b ensures conservative updates to prevent overfitting.<\/p>\n Below is an example dataset used to fine-tune an LLM using ranked preferences:<\/p>\n Each prompt has multiple responses with assigned ranks. The model learns to increase the probability of higher-ranked responses while reducing the probability of lower-ranked ones.<\/p>\n Below is the key function for computing the DPO loss and updating the model (only the main function is shown for scope; the full notebook is\u00a0linked<\/a>). The GRPO training function processes multiple ranked responses per prompt, optimizing log-likelihood differences while enforcing KL constraints.<\/p>\n The training loop processes ranked responses, computes loss, and updates the model while enforcing stability constraints.<\/p>\n \ud83d\udd17 Full implementation available here\u00a0<\/a><\/p>\n The expected outcomes of GRPO fine-tuning on the LLM, based on the provided outputs, highlight improvements in model optimization and preference-based ranking.<\/p>\n The training loss curve shows a gradual and stable decline over 15 epochs, indicating that the model is learning effectively. Unlike conventional policy optimization methods, GRPO ensures that ranked responses improve without drastic fluctuations, suggesting smooth convergence.<\/p>\n The loss value distribution over epochs presents a histogram where most values concentrate around a decreasing trend, showing that GRPO efficiently optimizes the model while maintaining stable loss updates. This distribution further indicates that loss values do not exhibit large variations, preventing instability in preference ranking.<\/p>\n The log probability distribution before vs. after fine-tuning provides crucial insights into the model\u2019s response generation. The shift in probability distribution suggests that after fine-tuning, the model assigns higher confidence to preferred responses. This shift results in responses that align better with human expectations and rankings.<\/p>\n Overall, the expected outcome of GRPO fine-tuning is a well-optimized model capable of generating high-quality responses ranked effectively based on preference learning. This demonstrates why GRPO is an effective alternative to traditional RL methods like PPO or DPO, offering a structured approach to optimizing LLMs without explicit reward models.<\/p>\n Unlike pairwise DPO and trust-region PPO, GRPO allows LLMs to learn from multiple ranked completions per prompt, significantly improving response quality, stability, and human alignment.<\/p>\n With reinforcement learning playing an increasingly central role in fine-tuning LLMs, GRPO stands out as the next step in AI preference learning, setting a new standard for human-aligned language modeling.<\/p>\n Policy optimization techniques play a critical role in reinforcement learning and LLM fine-tuning. Each method\u2014Policy Gradient (PG), Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO)\u2014offers unique advantages and trade-offs. PG serves as the foundation but suffers from high variance, while TRPO provides stability at the cost of computational complexity. PPO, being a refined version of TRPO, balances efficiency and robustness, making it widely used in RL applications. DPO, on the other hand, optimizes LLMs directly using preference data, eliminating the need for a reward model. Finally, GRPO, as introduced by DeepSeek, enhances preference-based fine-tuning by leveraging relative ranking in a structured manner.<\/p>\n Below is a comparison of these LLM Optimization methods based on key aspects such as variance, stability, sample efficiency, and suitability for reinforcement learning versus LLM fine-tuning:<\/p>\n For practitioners, the choice depends on the task at hand. If optimizing reinforcement learning agents in games or robotics, PPO is the best choice due to its balance of efficiency and performance. If high-stability optimization is required, TRPO is preferred despite its computational cost. DPO and GRPO, however, are better suited for LLM fine-tuning, with GRPO providing an even stronger optimization framework based on relative preference ranking rather than just binary preference signals.<\/p>\n Reinforcement learning (RL) plays a crucial role in both game-playing agents and LLM fine-tuning, but the optimization techniques vary significantly.<\/p>\n This blog highlights how reinforcement learning and preference-based fine-tuning are converging, with new techniques like GRPO bridging the gap between structured optimization and real-world deployment of large-scale AI systems.<\/p>\n The media shown in this article is not owned by Analytics Vidhya and is used at the Author\u2019s discretion.<\/strong><\/p>\n Ans. PPO (Proximal Policy Optimization) is an RL-based optimization method that improves policies while maintaining stability using a clipping mechanism. It is widely used in reinforcement learning tasks such as robotics and game-playing AI. DPO (Direct Preference Optimization), on the other hand, is designed specifically for LLM fine-tuning, directly optimizing the model based on human preferences without requiring an explicit reward model. DPO is simpler and more efficient for aligning language models with human intent.<\/p>\n<\/p><\/div>\nPrimer on Policy Optimization Techniques<\/h2>\n
What is Policy Optimization?<\/h3>\n
<\/figure>\n1.\u00a0Gradient-Based Optimization (Policy Gradient Methods)<\/h4>\n
\n
2.\u00a0Trust-Region Optimization (TRPO, PPO)<\/h4>\n
\n
3.\u00a0Preference-Based Optimization (DPO, GRPO)<\/h4>\n
\n
Mathematical Foundations (Required for All Methods)<\/h2>\n
A. Markov Decision Process (MDP)<\/h3>\n
<\/figure>\n\n
\u00a0 B. Expected Return J(\u03b8)<\/h3>\n
<\/figure>\nC. Policy Gradient Theorem<\/h3>\n
<\/figure>\n\n
D. Advantage Function A(s,a)<\/h3>\n
<\/figure>\n\n
Policy Gradient (PG) \u2013 The Foundation<\/h2>\n
The Policy Gradient Theorem<\/h2>\n
<\/figure>\n<\/figure>\n\n
Code Example: REINFORCE Algorithm<\/h2>\n
def train_policy_gradient(env, policy, optimizer, num_episodes=500, gamma=0.99):\n \"\"\"Train a policy using the REINFORCE algorithm\"\"\"\n reward_history = []\n\n for episode in range(num_episodes):\n state, _ = env.reset()\n log_probs = []\n rewards = []\n done = False\n\n while not done:\n state = torch.FloatTensor(state).unsqueeze(0)\n action_probs = policy(state)\n action_dist = torch.distributions.Categorical(action_probs)\n action = action_dist.sample()\n\n log_probs.append(action_dist.log_prob(action))\n next_state, reward, done, _, _ = env.step(action.item())\n rewards.append(reward)\n state = next_state\n\n # Compute discounted rewards\n returns = []\n G = 0\n for r in reversed(rewards):\n G = r + gamma * G\n returns.insert(0, G)\n\n returns = torch.tensor(returns)\n returns = (returns - returns.mean()) \/ (returns.std() + 1e-9) # Normalize for stability\n\n # Compute policy gradient loss\n loss = []\n for log_prob, G in zip(log_probs, returns):\n loss.append(-log_prob * G) # Gradient ascent on expected return\n loss = torch.stack(loss).sum()\n\n # Optimize policy\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n reward_history.append(sum(rewards))\n\n return reward_history<\/code><\/pre>\nCode Explanation<\/h3>\n
Expected Outcomes & Justification<\/h4>\n
<\/figure>\n<\/figure>\nTrust Region Policy Optimization (TRPO)\u00a0<\/h2>\n
<\/figure>\nTRPO Algorithm & Key Mathematical Concepts<\/h2>\n
<\/figure>\n<\/figure>\n<\/figure>\nCode Example: TRPO Training Loop<\/h2>\n
def train_trpo(env, policy, num_episodes=500, gamma=0.99):\n reward_history = []\n\n for episode in range(num_episodes):\n state = env.reset()\n if isinstance(state, tuple):\n state = state[0] # Handle Gym versions that return (state, info)\n\n log_probs = []\n states = []\n actions = []\n rewards = []\n\n done = False\n while not done:\n state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n probs = policy(state_tensor)\n action_dist = torch.distributions.Categorical(probs)\n action = action_dist.sample()\n\n step_result = env.step(action.item())\n\n if len(step_result) == 5:\n next_state, reward, terminated, truncated, _ = step_result\n done = terminated or truncated # New Gym API\n else:\n next_state, reward, done, _ = step_result # Old Gym API\n\n log_probs.append(action_dist.log_prob(action))\n states.append(state_tensor)\n actions.append(action)\n rewards.append(reward)\n\n state = next_state\n\n # Compute discounted rewards and advantages\n discounted_rewards = compute_discounted_rewards(rewards, gamma)\n discounted_rewards = (discounted_rewards - discounted_rewards.mean()) \n \/ (discounted_rewards.std() + 1e-9)\n\n # Convert lists to tensors\n states = torch.cat(states)\n actions = torch.tensor(actions)\n advantages = discounted_rewards\n\n # Copy old policy before updating\n old_policy = PolicyNetwork(env.observation_space.shape[0], \n env.action_space.n)\n old_policy.load_state_dict(policy.state_dict())\n\n # Apply TRPO update\n trpo_step(policy, old_policy, states, actions, advantages)\n\n total_episode_reward = sum(rewards)\n reward_history.append(total_episode_reward)\n\n if (episode + 1) % 50 == 0:\n print(f\"Episode {episode+1}, Total Reward: {total_episode_reward}\")\n\n return reward_history<\/code><\/pre>\nCode Explanation<\/h3>\n
Expected Outcomes & Justification<\/h3>\n
<\/figure>\n<\/figure>\nProximal Policy Optimization (PPO)<\/h2>\n
<\/figure>\n<\/figure>\n\n
PPO Algorithm & Key Mathematical Concept<\/h2>\n
<\/figure>\n<\/figure>\nCode Example: PPO Training Loop<\/h2>\n
def train_ppo(env, policy, optimizer, num_episodes=500, gamma=0.99, lambda_=0.95, epsilon=0.2, batch_size=32, epochs=5):\n reward_history = []\n\n for episode in range(num_episodes):\n state = env.reset()\n if isinstance(state, tuple):\n state = state[0] # Handle Gym versions returning (state, info)\n\n log_probs = []\n values = []\n states = []\n actions = []\n rewards = []\n\n done = False\n while not done:\n state_tensor = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n probs = policy(state_tensor)\n action_dist = torch.distributions.Categorical(probs)\n action = action_dist.sample()\n\n step_result = env.step(action.item())\n\n # Handle different Gym API versions\n if len(step_result) == 5:\n next_state, reward, terminated, truncated, _ = step_result\n done = terminated or truncated # New API\n else:\n next_state, reward, done, _ = step_result # Old API\n\n log_probs.append(action_dist.log_prob(action))\n states.append(state_tensor)\n actions.append(action)\n rewards.append(reward)\n\n state = next_state\n\n # Compute advantages\n values = [0] * len(rewards) # Placeholder for value estimates (since we use policy-only PPO)\n advantages = compute_advantages(rewards, values, gamma, lambda_)\n advantages = (advantages - advantages.mean()) \/ (advantages.std() + 1e-9) # Normalize advantages\n\n # Convert lists to tensors\n states = torch.cat(states)\n actions = torch.tensor(actions)\n old_log_probs = torch.tensor(log_probs)\n\n # PPO Training Loop\n for _ in range(epochs):\n for i in range(0, len(states), batch_size):\n batch_indices = slice(i, i + batch_size)\n\n new_probs = policy(states[batch_indices])\n new_action_dist = torch.distributions.Categorical(new_probs)\n new_log_probs = new_action_dist.log_prob(actions[batch_indices])\n\n loss = ppo_loss(old_log_probs[batch_indices], new_log_probs, advantages[batch_indices], epsilon)\n\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n total_episode_reward = sum(rewards)\n reward_history.append(total_episode_reward)\n\n if (episode + 1) % 50 == 0:\n print(f\"Episode {episode+1}, Total Reward: {total_episode_reward}\")\n\n return reward_history<\/code><\/pre>\nCode Explanation<\/h3>\n
\n
Expected Outcomes for PPO<\/h3>\n
<\/figure>\n<\/figure>\nKey Observations:<\/h4>\n
\n
\n
Direct Preference Optimization (DPO) \u2013 Preference Learning for LLMs<\/h2>\n
The DPO Training Dataset<\/h3>\n
preference_data = [\n {\"prompt\": \"What is the capital of France?\",\n \"preferred\": \"The capital of France is Paris.\",\n \"rejected\": \"France is a country in Europe.\"},\n\n {\"prompt\": \"Who wrote Hamlet?\",\n \"preferred\": \"Hamlet was written by William Shakespeare.\",\n \"rejected\": \"Hamlet is an old book.\"},\n\n {\"prompt\": \"Tell me a joke.\",\n \"preferred\": \"Why did the scarecrow win an award? Because he was outstanding in his field!\",\n \"rejected\": \"I don\u2019t know any jokes.\"},\n\n {\"prompt\": \"What is artificial intelligence?\",\n \"preferred\": \"Artificial intelligence is the simulation of human intelligence in machines.\",\n \"rejected\": \"AI is just robots.\"},\n\n {\"prompt\": \"How to stay motivated?\",\n \"preferred\": \"Set clear goals, track progress, and reward yourself for achievements.\",\n \"rejected\": \"Just be motivated.\"},\n]\n<\/code><\/pre>\nThe DPO Loss Function<\/h3>\n
<\/figure>\n\n
Code Example: Direct Preference Optimization (DPO)<\/h2>\n
def dpo_loss(preferred_log_probs, rejected_log_probs, beta=0.1):\n \"\"\"Computes the DPO loss function to optimize based on preferences\"\"\"\n return -torch.mean(torch.sigmoid(beta * (preferred_log_probs - \n rejected_log_probs)))\n\ndef encode_text(prompt, response):\n \"\"\"Encodes the prompt + response into tokenized format with proper padding\"\"\"\n tokenizer.pad_token = tokenizer.eos_token # Fix padding issue\n input_text = f\"User: {prompt}\\nAssistant: {response}\"\n\n inputs = tokenizer(\n input_text,\n return_tensors=\"pt\",\n padding=True, # Enable padding\n truncation=True, # Truncate if too long\n max_length=512 # Set max length for safety\n )\n\n return inputs[\"input_ids\"], inputs[\"attention_mask\"]\n\nloss_history = [] # Store loss values\n\noptimizer = optim.AdamW(model.parameters(), lr=5e-5)\n\nfor epoch in range(10): # Train for 10 epochs\n total_loss = 0\n\n for data in preference_data:\n prompt, preferred, rejected = data[\"prompt\"], data[\"preferred\"], \n data[\"rejected\"]\n\n # Encode preferred and rejected responses\n pref_input_ids, pref_attention_mask = encode_text(prompt, preferred)\n rej_input_ids, rej_attention_mask = encode_text(prompt, rejected)\n\n # Get log probabilities from the model\n preferred_logits = model(pref_input_ids, attention_mask=\n pref_attention_mask).logits[:, -1, :]\n rejected_logits = model(rej_input_ids, attention_mask=rej_attention_mask)\n .logits[:, -1, :]\n\n preferred_log_probs = preferred_logits.log_softmax(dim=-1)\n rejected_log_probs = rejected_logits.log_softmax(dim=-1)\n\n # Compute DPO loss\n loss = dpo_loss(preferred_log_probs, rejected_log_probs, beta=0.5)\n\n # Optimize the model\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n\n total_loss += loss.item()\n\n loss_history.append(total_loss) # Store loss for visualization\n print(f\"Epoch {epoch + 1}, Loss: {total_loss:.4f}\")\n<\/code><\/pre>\nExpected Output & Analysis<\/h3>\n
<\/figure>\n<\/figure>\n<\/figure>\n<\/figure>\nGRPO \u2013 Group Relative Policy Optimization (DeepSeek\u2019s Approach)<\/h2>\n
Mathematical Foundation of GRPO<\/h2>\n
1. Expected Return in Preference Optimization<\/h3>\n
<\/figure>\n2. Ranking-Based Probability Optimization<\/h3>\n
<\/figure>\n<\/figure>\n\n
<\/figure>\nData for GRPO Fine-Tuning<\/h2>\n
grpo_preference_data = [\n {\"prompt\": \"What is the capital of France?\",\n \"responses\": [\n {\"text\": \"The capital of France is Paris.\", \"rank\": 1},\n {\"text\": \"Paris is the largest city in France.\", \"rank\": 2},\n {\"text\": \"Paris is in France.\", \"rank\": 3},\n {\"text\": \"France is a country in Europe.\", \"rank\": 4}\n ]},\n\n {\"prompt\": \"Tell me a joke.\",\n \"responses\": [\n {\"text\": \"Why did the scarecrow win an award? Because he was outstanding \n in his field!\", \"rank\": 1},\n {\"text\": \"Why did the chicken cross the road? To get to the other side.\",\n \"rank\": 2},\n {\"text\": \"Jokes are funny.\", \"rank\": 3},\n {\"text\": \"I don\u2019t know any jokes.\", \"rank\": 4}\n ]}\n]<\/code><\/pre>\nCode Implementation: Group-Based Preference Optimization<\/h2>\n
def deepseek_grpo_loss(log_probs, rankings, input_ids, beta=1.0, kl_penalty=0.02, epsilon=1e-6):\n \"\"\"Computes DeepSeek GRPO loss with pairwise ranking and KL regularization.\"\"\"\n loss_terms = []\n num_pairs = 0\n\n log_probs = torch.clamp(log_probs, min=-10, max=10) # Prevent extreme values\n\n for i in range(len(rankings)):\n for j in range(i + 1, len(rankings)):\n if rankings[i] 0 else torch.tensor(0.0, device=log_probs.device)\n\n # KL regularization to prevent policy divergence\n old_logits = base_model(input_ids).logits[:, -1, :]\n old_log_probs = old_logits.log_softmax(dim=-1)\n\n kl_div = torch.nn.functional.kl_div(log_probs, old_log_probs.clamp(min=epsilon), reduction=\"batchmean\")\n\n return loss + (kl_penalty * kl_div.mean()) # Ensure single scalar<\/code><\/pre>\nTraining Loop for GRPO<\/h2>\n
loss_history = []\nnum_epochs = 15\n\nfor epoch in range(num_epochs):\n total_loss = 0\n\n for data in grpo_preference_data:\n prompt, responses = data[\"prompt\"], data[\"responses\"]\n\n input_ids, rankings = encode_text(prompt, responses)\n\n logits = model(input_ids).logits[:, -1, :]\n log_probs = logits.log_softmax(dim=-1)\n\n loss = deepseek_grpo_loss(log_probs, rankings, input_ids)\n\n if torch.isnan(loss):\n print(f\"Skipping update at epoch {epoch} due to NaN loss.\")\n continue\n\n optimizer.zero_grad()\n loss.backward()\n\n torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n\n optimizer.step()\n total_loss += loss.item()\n\n loss_history.append(total_loss)\n scheduler.step()\n print(f\"Epoch {epoch + 1}, Loss: {total_loss:.4f}\")\n<\/code><\/pre>\nExpected Outcome and Results<\/h2>\n
<\/figure>\n<\/figure>\n<\/figure>\n<\/figure>\nFinal Model Insights: Why GRPO Excels in LLM Fine-Tuning<\/h2>\n
\n
Conclusion<\/h2>\n
\n
\n Method<\/th>\n Variance<\/th>\n Stability<\/th>\n Sample Efficiency<\/th>\n Best for<\/th>\n Limitations<\/th>\n<\/tr>\n \n PG (REINFORCE)<\/td>\n High<\/td>\n Low<\/td>\n Inefficient<\/td>\n Simple RL problems<\/td>\n High variance, slow convergence<\/td>\n<\/tr>\n \n TRPO<\/td>\n Low<\/td>\n High<\/td>\n Moderate<\/td>\n High-stability RL tasks<\/td>\n Complex second-order updates, expensive<\/td>\n<\/tr>\n \n PPO<\/td>\n Medium<\/td>\n High<\/td>\n Efficient<\/td>\n General RL tasks, Robotics, Games<\/td>\n May require careful hyperparameter tuning<\/td>\n<\/tr>\n \n DPO<\/td>\n Low<\/td>\n High<\/td>\n High<\/td>\n LLM fine-tuning with human preferences<\/td>\n Lacks explicit reinforcement learning framework<\/td>\n<\/tr>\n \n GRPO<\/td>\n Low<\/td>\n High<\/td>\n High<\/td>\n Preference-based LLM fine-tuning<\/td>\n Newer method, requires further empirical validation<\/td>\n<\/tr>\n<\/table>\n Key Takeaways<\/h3>\n
\n
Frequently Asked Questions<\/h2>\n
![\"Neil](\"https:\/\/av-eks-lekhak.s3.amazonaws.com\/media\/lekhak-profile-images\/converted_image_sT5MuqV.webp\")
<\/p>\n