Chapter 2: Fundamental Theory of Reinforcement Learning

This chapter organizes the intuition behind representative methods such as Q-learning, DQN, and PPO, along with their differences. You will develop a sense of which method to try for which type of problem.

💡 Supplement: Continuous control pairs well with policy-gradient methods such as PPO. For discrete choices, start with Q-based methods.

Learning Objectives

In this chapter, you will master the following:

• Theory and implementation of Policy Gradient Methods
• The mechanism of the Actor-Critic architecture
• Details of Proximal Policy Optimization (PPO)
• Practical implementation with Stable Baselines3

2.1 Policy Gradient Methods

Limitations of Q-Learning

The Q-learning and DQN methods from Chapter 1 were value-based methods. These have the following limitations:

1. Discrete actions only: $\arg\max_a Q(s,a)$ is difficult in continuous action spaces
2. Deterministic policy: Always selects the same action (cannot learn a stochastic policy)
3. Fragile to small changes: A tiny change in the Q-value can drastically change the policy

In materials science, continuous control (raising the temperature by 0.5 degrees, changing the composition ratio by 2%) is important.

Basic Idea of Policy Gradient Methods

Policy gradient methods optimize the policy directly:

$$ \pi_\theta(a|s) = P(a|s; \theta) $$

• $\theta$: policy parameters (weights of the neural network)

Objective: Maximize the expected cumulative reward $J(\theta)$

$$ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^T r_t \right] $$

• $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$: trajectory

Policy Gradient Theorem

The REINFORCE algorithm (Williams, 1992) computes the gradient as follows:

$$ \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot R_t \right] $$

• $R_t = \sum_{k=t}^T \gamma^{k-t} r_k$: cumulative reward (return) from time $t$

Intuitive meaning: - Increase the probability of actions that received high rewards - Decrease the probability of actions that received low rewards

Implementation of REINFORCE

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

class PolicyNetwork(nn.Module):
    """Policy network

    Takes a state as input and outputs the probability of each action
    """
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(PolicyNetwork, self).__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, action_dim)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return torch.softmax(self.fc3(x), dim=-1)  # Probability distribution


class REINFORCEAgent:
    """REINFORCE algorithm"""
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99):
        self.gamma = gamma
        self.policy = PolicyNetwork(state_dim, action_dim)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=lr)

        # Store the log within the episode
        self.log_probs = []
        self.rewards = []

    def select_action(self, state):
        """Sample an action according to the policy"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0)
        probs = self.policy(state_tensor)

        # Sample from the probability distribution
        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample()

        # Store the log probability (for gradient computation)
        self.log_probs.append(action_dist.log_prob(action))

        return action.item()

    def store_reward(self, reward):
        """Store the reward"""
        self.rewards.append(reward)

    def update(self):
        """Update the policy after the episode ends"""
        # Compute the returns (cumulative rewards)
        returns = []
        R = 0
        for r in reversed(self.rewards):
            R = r + self.gamma * R
            returns.insert(0, R)

        returns = torch.FloatTensor(returns)

        # Normalize (stabilize learning)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)

        # Policy gradient
        policy_loss = []
        for log_prob, R in zip(self.log_probs, returns):
            policy_loss.append(-log_prob * R)

        # Gradient descent
        self.optimizer.zero_grad()
        loss = torch.stack(policy_loss).sum()
        loss.backward()
        self.optimizer.step()

        # Reset the log
        self.log_probs = []
        self.rewards = []


# Simple materials exploration environment (discrete action version)
class DiscreteMaterialsEnv:
    """Discrete-action materials exploration environment"""
    def __init__(self, state_dim=4):
        self.state_dim = state_dim
        self.target = np.array([3.0, 5.0, 2.5, 4.0])
        self.state = None

    def reset(self):
        self.state = np.random.uniform(0, 10, self.state_dim)
        return self.state

    def step(self, action):
        # Actions: 0=increase dim 0, 1=decrease dim 0, 2=increase dim 1, 3=decrease dim 1
        dim = action // 2
        delta = 0.5 if action % 2 == 0 else -0.5

        self.state[dim] = np.clip(self.state[dim] + delta, 0, 10)

        # Reward: distance to the target
        distance = np.linalg.norm(self.state - self.target)
        reward = -distance

        done = distance < 0.5

        return self.state, reward, done


# Training REINFORCE
env = DiscreteMaterialsEnv()
agent = REINFORCEAgent(state_dim=4, action_dim=4)

episodes = 1000
rewards_history = []

for episode in range(episodes):
    state = env.reset()
    total_reward = 0
    done = False

    while not done:
        action = agent.select_action(state)
        next_state, reward, done = env.step(action)

        agent.store_reward(reward)
        state = next_state
        total_reward += reward

    # Update after the episode ends
    agent.update()
    rewards_history.append(total_reward)

    if (episode + 1) % 100 == 0:
        avg_reward = np.mean(rewards_history[-100:])
        print(f"Episode {episode+1}: Avg Reward = {avg_reward:.2f}")

# Learning curve
plt.figure(figsize=(10, 6))
plt.plot(np.convolve(rewards_history, np.ones(20)/20, mode='valid'))
plt.xlabel('Episode')
plt.ylabel('Average Reward (20 episodes)')
plt.title('REINFORCE: Materials Exploration with Policy Gradient')
plt.grid(True)
plt.show()

Example output:

Episode 100: Avg Reward = -38.24
Episode 200: Avg Reward = -28.15
Episode 500: Avg Reward = -15.32
Episode 1000: Avg Reward = -7.89

2.2 Baselines and Variance Reduction

Problems with REINFORCE

REINFORCE has high variance. Even with the same policy, the return $R_t$ fluctuates greatly depending on whether luck is good or bad.

Introducing a Baseline

By subtracting a baseline $b(s)$, the variance is reduced:

$$ \nabla_\theta J(\theta) = \mathbb{E} \left[ \sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot (R_t - b(s_t)) \right] $$

Optimal baseline: the state-value function $V(s)$

$$ b(s_t) = V(s_t) = \mathbb{E}_{\pi} \left[ \sum_{k=t}^T \gamma^{k-t} r_k \mid s_t \right] $$

Advantage function $A(s, a)$: $$ A(s, a) = Q(s, a) - V(s) = R_t - V(s_t) $$

This represents "how much better this action is than average."

REINFORCE with a Baseline

class ValueNetwork(nn.Module):
    """Value network (baseline)"""
    def __init__(self, state_dim, hidden_dim=64):
        super(ValueNetwork, self).__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, 1)  # Output the state value

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return self.fc3(x)


class REINFORCEWithBaseline:
    """REINFORCE with a baseline"""
    def __init__(self, state_dim, action_dim, lr=1e-3, gamma=0.99):
        self.gamma = gamma
        self.policy = PolicyNetwork(state_dim, action_dim)
        self.value = ValueNetwork(state_dim)

        self.policy_optimizer = optim.Adam(self.policy.parameters(), lr=lr)
        self.value_optimizer = optim.Adam(self.value.parameters(), lr=lr)

        self.log_probs = []
        self.rewards = []
        self.states = []

    def select_action(self, state):
        state_tensor = torch.FloatTensor(state).unsqueeze(0)
        probs = self.policy(state_tensor)

        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample()

        self.log_probs.append(action_dist.log_prob(action))
        self.states.append(state)

        return action.item()

    def store_reward(self, reward):
        self.rewards.append(reward)

    def update(self):
        # Compute the returns
        returns = []
        R = 0
        for r in reversed(self.rewards):
            R = r + self.gamma * R
            returns.insert(0, R)

        returns = torch.FloatTensor(returns)
        states = torch.FloatTensor(self.states)

        # Output of the value network (baseline)
        values = self.value(states).squeeze()

        # Advantage = return - baseline
        advantages = returns - values.detach()

        # Policy gradient loss
        policy_loss = []
        for log_prob, adv in zip(self.log_probs, advantages):
            policy_loss.append(-log_prob * adv)

        # Value network loss (MSE)
        value_loss = nn.MSELoss()(values, returns)

        # Optimization
        self.policy_optimizer.zero_grad()
        torch.stack(policy_loss).sum().backward()
        self.policy_optimizer.step()

        self.value_optimizer.zero_grad()
        value_loss.backward()
        self.value_optimizer.step()

        # Reset
        self.log_probs = []
        self.rewards = []
        self.states = []


# Training (with a baseline)
agent_baseline = REINFORCEWithBaseline(state_dim=4, action_dim=4)

rewards_baseline = []
for episode in range(1000):
    state = env.reset()
    total_reward = 0
    done = False

    while not done:
        action = agent_baseline.select_action(state)
        next_state, reward, done = env.step(action)

        agent_baseline.store_reward(reward)
        state = next_state
        total_reward += reward

    agent_baseline.update()
    rewards_baseline.append(total_reward)

# Comparison
plt.figure(figsize=(10, 6))
plt.plot(np.convolve(rewards_history, np.ones(20)/20, mode='valid'), label='REINFORCE')
plt.plot(np.convolve(rewards_baseline, np.ones(20)/20, mode='valid'), label='REINFORCE + Baseline')
plt.xlabel('Episode')
plt.ylabel('Average Reward')
plt.title('Learning Stabilization via Baseline')
plt.legend()
plt.grid(True)
plt.show()

Result: With the baseline, learning becomes more stable and convergence becomes faster.

2.3 Actor-Critic Architecture

Concept of Actor-Critic

Learn the Actor (policy) and the Critic (value function) simultaneously:

• Actor $\pi_\theta(a|s)$: selects the action
• Critic $V_\phi(s)$: evaluates the value of the state

TD Error and Advantage

TD error (Temporal Difference Error): $$ \delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t) $$

This can be used as a one-step advantage estimate.

A2C (Advantage Actor-Critic)

class A2CAgent:
    """Advantage Actor-Critic"""
    def __init__(self, state_dim, action_dim, lr_actor=1e-3, lr_critic=1e-3, gamma=0.99):
        self.gamma = gamma

        self.actor = PolicyNetwork(state_dim, action_dim)
        self.critic = ValueNetwork(state_dim)

        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=lr_actor)
        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=lr_critic)

    def select_action(self, state):
        state_tensor = torch.FloatTensor(state).unsqueeze(0)
        probs = self.actor(state_tensor)

        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample()

        return action.item(), action_dist.log_prob(action)

    def update(self, state, action_log_prob, reward, next_state, done):
        """Update at every step"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0)
        next_state_tensor = torch.FloatTensor(next_state).unsqueeze(0)

        # Current and next state values
        value = self.critic(state_tensor)
        next_value = self.critic(next_state_tensor)

        # TD target and TD error
        td_target = reward + (1 - done) * self.gamma * next_value.item()
        td_error = td_target - value.item()

        # Critic loss (MSE)
        critic_loss = (torch.FloatTensor([td_target]) - value).pow(2)

        # Actor loss (policy gradient x advantage)
        actor_loss = -action_log_prob * td_error

        # Optimization
        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

        self.critic_optimizer.zero_grad()
        critic_loss.backward()
        self.critic_optimizer.step()


# A2C training
agent_a2c = A2CAgent(state_dim=4, action_dim=4)

rewards_a2c = []
for episode in range(1000):
    state = env.reset()
    total_reward = 0
    done = False

    while not done:
        action, log_prob = agent_a2c.select_action(state)
        next_state, reward, done = env.step(action)

        # Update at every step
        agent_a2c.update(state, log_prob, reward, next_state, done)

        state = next_state
        total_reward += reward

    rewards_a2c.append(total_reward)

    if (episode + 1) % 100 == 0:
        avg_reward = np.mean(rewards_a2c[-100:])
        print(f"Episode {episode+1}: Avg Reward = {avg_reward:.2f}")

Advantages: - Online learning without waiting for the episode to end - Low variance thanks to the TD error

2.4 Proximal Policy Optimization (PPO)

Trust Region Methods

In policy gradient methods, if the update is too large, the policy can collapse.

Trust Region Policy Optimization (TRPO) constrains the change in the policy:

$$ \max_\theta \mathbb{E} \left[ \frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A(s, a) \right] \quad \text{s.t.} \quad D_{\text{KL}}(\pi_{\theta_{\text{old}}} | \pi_\theta) \leq \delta $$

However, optimizing with a KL divergence constraint is complex.

Simplification by PPO

PPO (Schulman et al., 2017) realizes the constraint through clipping within the loss function:

$$ L^{\text{CLIP}}(\theta) = \mathbb{E} \left[ \min \left( r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t \right) \right] $$

• $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$: importance ratio
• $\epsilon$: clipping range (usually 0.1 to 0.2)

Intuition: - Advantage is positive (good action) -> increase $r_t$, but cap it at $1+\epsilon$ - Advantage is negative (bad action) -> decrease $r_t$, but floor it at $1-\epsilon$ - Prevent abrupt policy changes

Entropy Bonus

To encourage exploration, add entropy to the loss:

$$ L^{\text{PPO}}(\theta) = L^{\text{CLIP}}(\theta) + c_1 L^{\text{VF}}(\theta) - c_2 H[\pi_\theta] $$

• $L^{\text{VF}}$: value function loss
• $H[\pi_\theta] = -\sum_a \pi_\theta(a|s) \log \pi_\theta(a|s)$: entropy (uncertainty of the probability distribution)
• $c_2$: entropy coefficient (usually 0.01)

Implementation of PPO (using Stable Baselines3)

from stable_baselines3 import PPO
from stable_baselines3.common.vec_env import DummyVecEnv
import gym

# Gym environment wrapper
class GymMaterialsEnv(gym.Env):
    """OpenAI Gym-compatible materials exploration environment"""
    def __init__(self):
        super(GymMaterialsEnv, self).__init__()
        self.state_dim = 4
        self.target = np.array([3.0, 5.0, 2.5, 4.0])

        # Define the action and state spaces
        self.action_space = gym.spaces.Discrete(4)
        self.observation_space = gym.spaces.Box(
            low=0, high=10, shape=(self.state_dim,), dtype=np.float32
        )

        self.state = None

    def reset(self):
        self.state = np.random.uniform(0, 10, self.state_dim).astype(np.float32)
        return self.state

    def step(self, action):
        dim = action // 2
        delta = 0.5 if action % 2 == 0 else -0.5

        self.state[dim] = np.clip(self.state[dim] + delta, 0, 10)

        distance = np.linalg.norm(self.state - self.target)
        reward = -distance
        done = distance < 0.5

        return self.state, reward, done, {}

    def render(self, mode='human'):
        pass


# Create the environment
env = DummyVecEnv([lambda: GymMaterialsEnv()])

# PPO model
model = PPO(
    "MlpPolicy",                # Multilayer perceptron policy
    env,
    learning_rate=3e-4,
    n_steps=2048,               # Number of steps before an update
    batch_size=64,
    n_epochs=10,                # Number of optimization epochs per update
    gamma=0.99,
    gae_lambda=0.95,            # GAE (Generalized Advantage Estimation)
    clip_range=0.2,             # PPO clipping range
    ent_coef=0.01,              # Entropy coefficient
    verbose=1,
    tensorboard_log="./ppo_materials_tensorboard/"
)

# Training
model.learn(total_timesteps=100000)

# Save
model.save("ppo_materials_agent")

# Evaluation
eval_env = GymMaterialsEnv()
state = eval_env.reset()
total_reward = 0

for _ in range(100):
    action, _ = model.predict(state, deterministic=True)
    state, reward, done, _ = eval_env.step(action)
    total_reward += reward

    if done:
        break

print(f"Evaluation result: Total Reward = {total_reward:.2f}")
print(f"Final state: {state}")
print(f"Target: {eval_env.target}")

Example output:

---------------------------------
| rollout/           |          |
|    ep_len_mean     | 45.2     |
|    ep_rew_mean     | -15.3    |
| time/              |          |
|    fps             | 1024     |
|    iterations      | 50       |
|    time_elapsed    | 97       |
|    total_timesteps | 102400   |
---------------------------------

Evaluation result: Total Reward = -5.23
Final state: [3.02 4.98 2.47 3.95]
Target: [3.  5.  2.5 4. ]

Explanation: - With Stable Baselines3, PPO is implemented in just a few lines - Learning progress can be visualized with TensorBoard - A material very close to the target is discovered

2.5 Extension to Continuous Action Spaces

Gaussian Policy

In materials science, continuous control of temperature, composition ratios, and so on is required.

For continuous actions, use a Gaussian distribution policy:

$$ \pi_\theta(a|s) = \mathcal{N}(\mu_\theta(s), \sigma_\theta(s)) $$

• $\mu_\theta(s)$: mean (output by the neural network)
• $\sigma_\theta(s)$: standard deviation (learnable or fixed)

Continuous-Action PPO

# Continuous-action environment
class ContinuousGymMaterialsEnv(gym.Env):
    """Continuous-action materials exploration environment"""
    def __init__(self):
        super(ContinuousGymMaterialsEnv, self).__init__()
        self.state_dim = 4
        self.target = np.array([3.0, 5.0, 2.5, 4.0])

        # Continuous action space (4-dimensional vector, range [-1, 1])
        self.action_space = gym.spaces.Box(
            low=-1, high=1, shape=(self.state_dim,), dtype=np.float32
        )
        self.observation_space = gym.spaces.Box(
            low=0, high=10, shape=(self.state_dim,), dtype=np.float32
        )

        self.state = None

    def reset(self):
        self.state = np.random.uniform(0, 10, self.state_dim).astype(np.float32)
        return self.state

    def step(self, action):
        # Map the action to a state change (-1 to 1 -> -0.5 to 0.5)
        delta = action * 0.5
        self.state = np.clip(self.state + delta, 0, 10)

        distance = np.linalg.norm(self.state - self.target)
        reward = -distance
        done = distance < 0.3

        return self.state, reward, done, {}

    def render(self, mode='human'):
        pass


# Continuous-action PPO
env_continuous = DummyVecEnv([lambda: ContinuousGymMaterialsEnv()])

model_continuous = PPO(
    "MlpPolicy",
    env_continuous,
    learning_rate=3e-4,
    n_steps=2048,
    batch_size=64,
    n_epochs=10,
    gamma=0.99,
    clip_range=0.2,
    verbose=1
)

model_continuous.learn(total_timesteps=100000)

# Evaluation
eval_env_cont = ContinuousGymMaterialsEnv()
state = eval_env_cont.reset()

for _ in range(50):
    action, _ = model_continuous.predict(state, deterministic=True)
    state, reward, done, _ = eval_env_cont.step(action)

    if done:
        break

print(f"Final state: {state}")
print(f"Target: {eval_env_cont.target}")
print(f"Distance: {np.linalg.norm(state - eval_env_cont.target):.4f}")

Example output:

Final state: [3.001 5.003 2.498 3.997]
Target: [3.  5.  2.5 4. ]
Distance: 0.0054

Explanation: Continuous actions enable precise control toward the target.

Exercises

Problem 1 (Difficulty: easy)

Explain why using a baseline reduces the variance, using the following expression.

$$ \text{Var}[R_t] \quad \text{vs.} \quad \text{Var}[R_t - b(s_t)] $$

Problem 2 (Difficulty: medium)

Explain what happens when the PPO clipping range $\epsilon$ is increased, and what happens in the extreme case of $\epsilon=0$.

# ε=0.05 (tight constraint)
model_tight = PPO("MlpPolicy", env, clip_range=0.05)

# ε=0.5 (loose constraint)
model_loose = PPO("MlpPolicy", env, clip_range=0.5)

# Compare the learning curves
# -> model_tight is stable but slow
# -> model_loose is fast but oscillates

Problem 3 (Difficulty: hard)

In materials exploration, compare the following two reward designs and describe the strengths and weaknesses of each. Also, actually experiment with them in code.

Reward A (sparse reward): reward 1 only when the target is reached, 0 otherwise Reward B (dense reward): a continuous reward based on the distance to the target

# Reward A (sparse reward)
class SparseRewardEnv(gym.Env):
    def step(self, action):
        # ... (state update) ...
        distance = np.linalg.norm(self.state - self.target)

        if distance < 0.5:
            reward = 1.0  # Reached
            done = True
        else:
            reward = 0.0  # Otherwise
            done = False

        return self.state, reward, done, {}

# Reward B (dense reward)
class DenseRewardEnv(gym.Env):
    def step(self, action):
        # ... (state update) ...
        distance = np.linalg.norm(self.state - self.target)
        reward = -distance  # Continuous reward
        done = distance < 0.5

        return self.state, reward, done, {}

# Comparison experiment
model_sparse = PPO("MlpPolicy", DummyVecEnv([lambda: SparseRewardEnv()]))
model_dense = PPO("MlpPolicy", DummyVecEnv([lambda: DenseRewardEnv()]))

model_sparse.learn(total_timesteps=100000)
model_dense.learn(total_timesteps=100000)

# Result: model_dense learns faster, but
# in complex environments model_sparse may find a better solution

Summary of This Section

• Policy gradient methods optimize the policy directly and handle continuous actions
• The REINFORCE algorithm has high variance, but is improved by a baseline
• Actor-Critic learns the Actor and Critic simultaneously, achieving low variance and online learning
• PPO achieves stable learning through clipping, and is a state-of-the-art practical method
• With Stable Baselines3, PPO can be implemented in just a few lines
• For continuous action spaces, use a Gaussian policy

In the next chapter, you will learn how to build custom environments specialized for materials exploration and how to design rewards.

Quality Checklist: Verifying Your Policy Gradient Implementation

Theoretical Understanding Skills

• [ ] Can explain the policy gradient theorem with equations
• [ ] Can derive the REINFORCE update equation
• [ ] Can explain why a baseline reduces variance
• [ ] Understands the role of PPO clipping

Implementation Skills

• [ ] Can implement a policy network in PyTorch
• [ ] Can compute the return (cumulative reward)
• [ ] Can compute the advantage function
• [ ] Can use PPO with Stable Baselines3

Application to Materials Exploration

• [ ] Can design continuous control variables such as temperature and pressure as an action space
• [ ] Can appropriately weight multi-objective rewards (yield, selectivity)
• [ ] Can incorporate safety constraints (such as temperature upper limits) into the reward

Debugging Skills

• [ ] Knows how to handle cases where the policy gradient variance is large
• [ ] Can identify the cause when PPO does not converge
• [ ] Can tune the entropy bonus

References

1. Williams "Simple statistical gradient-following algorithms for connectionist reinforcement learning" Machine Learning (1992) - REINFORCE
2. Mnih et al. "Asynchronous methods for deep reinforcement learning" ICML (2016) - A3C/A2C
3. Schulman et al. "Proximal policy optimization algorithms" arXiv (2017) - PPO
4. Schulman et al. "Trust region policy optimization" ICML (2015) - TRPO
5. Raffin et al. "Stable-Baselines3: Reliable reinforcement learning implementations" JMLR (2021)

Next Chapter: Chapter 3: Building Materials Exploration Environments