Skip to content

PPOTrainSettings

Full path: schola.scripts.sb3.train.settings.PPOTrainSettings

Dataclass for configuring the settings of the Proximal Policy Optimization (PPO) algorithm. This includes parameters for the learning process, such as learning rate, batch size, number of steps, and other hyperparameters that control the behavior of the PPO algorithm.

PPOTrainSettings(learning_rate = 0.0003, n_steps = 2048, batch_size = 64,
                 n_epochs = 10, gamma = 0.99, gae_lambda = 0.95,
                 clip_range = 0.2, normalize_advantage = True, ent_coef = 0.0,
                 vf_coef = 0.5, max_grad_norm = 0.5, use_sde = False,
                 sde_sample_freq = -1)

Bases: BaseSb3AlgorithmSettings, BasePPOSettings

Parameters

  • learning_rate (Annotated)

  • n_steps (Annotated)

  • batch_size (Annotated)

  • n_epochs (Annotated)

  • gamma (Annotated)

  • gae_lambda (Annotated)

  • clip_range (Annotated)

  • normalize_advantage (bool)

  • ent_coef (Annotated)

  • vf_coef (Annotated)

  • max_grad_norm (Annotated)

  • use_sde (bool)

  • sde_sample_freq (int)

Methods

init

__init__(learning_rate = 0.0003, n_steps = 2048, batch_size = 64, n_epochs = 10,
         gamma = 0.99, gae_lambda = 0.95, clip_range = 0.2,
         normalize_advantage = True, ent_coef = 0.0, vf_coef = 0.5,
         max_grad_norm = 0.5, use_sde = False, sde_sample_freq = -1)

Parameters

  • learning_rate (Annotated)

  • n_steps (Annotated)

  • batch_size (Annotated)

  • n_epochs (Annotated)

  • gamma (Annotated)

  • gae_lambda (Annotated)

  • clip_range (Annotated)

  • normalize_advantage (bool)

  • ent_coef (Annotated)

  • vf_coef (Annotated)

  • max_grad_norm (Annotated)

  • use_sde (bool)

  • sde_sample_freq (int)

Attributes

clip_range

clip_range

Clipping range for the policy update. This is the maximum amount by which the new policy can differ from the old policy during training. This helps to prevent large updates that can destabilize training.

ent_coef

ent_coef

Coefficient for the entropy term in the loss function. This encourages exploration by adding a penalty for certainty in the policy’s action distribution. A higher value will encourage more exploration, while a lower value will make the policy more deterministic. Set to 0.0 to disable entropy regularization.

gae_lambda

gae_lambda

Lambda parameter for Generalized Advantage Estimation (GAE). This parameter helps to balance bias and variance in the advantage estimation. A value of 1.0 corresponds to standard advantage estimation, while lower values will reduce variance but may introduce bias.

gamma

gamma

Discount factor for future rewards. This determines how much the agent values future rewards compared to immediate rewards. A value of 0.99 means that future rewards are discounted by 1% per time step.

max_grad_norm

max_grad_norm

Maximum gradient norm for clipping. This is used to prevent exploding gradients by scaling down the gradients if their norm exceeds this value. This can help to stabilize training, especially in environments with high variance in the rewards or gradients.

n_epochs

n_epochs

Number of epochs to update the policy. This is the number of times the model will iterate over the collected data during training. More epochs can lead to better convergence but also overfitting.

normalize_advantage

normalize_advantage

Whether to normalize the advantages. Normalizing the advantages can help to stabilize training by ensuring that they have a mean of 0 and a standard deviation of 1. This can lead to more consistent updates to the policy.

sde_sample_freq

sde_sample_freq

Frequency at which to sample the SDE noise. This determines how often the noise is sampled when using State Dependent Exploration (SDE). A value of -1 means that it will sample the noise at every step, while a positive integer will specify the number of steps between samples. This can help to control the exploration behavior of the agent.

use_sde

use_sde

Whether to use State Dependent Exploration (SDE). This can help to improve exploration by adapting the exploration noise based on the current state of the environment. When set to True, it will use SDE for exploration instead of the standard exploration strategy.

vf_coef

vf_coef

Coefficient for the value function loss in the overall loss function. This determines how much weight is given to the value function loss compared to the policy loss. A higher value will put more emphasis on accurately estimating the value function, while a lower value will prioritize the policy update.