Example 2: Ball Shooter

The Structure of the Environment in Unreal Engine

To build the game (called environment hereafter) where the agent will learn to shoot moving targets, we need the following in our Unreal Engine project:

• Map: The game map includes the floor, four walls, the agent, and the environment definition.

• Ball Blueprint: The projectile that the agent shoots, which is spawned by the agent when it takes the shoot action.

• Target Blueprint: The object that the agent will shoot, which moves randomly around the map and is destroyed when hit three times by a ball.

• Agent blueprint: A subclass of Character, which includes the shape and appearance of the agent.

• Shooting Actuator: A custom discrete Actuator that allows the agent to shoot the ball.

• Discrete Rotation Actuator: A custom discrete Actuator that allows the agent to rotate.

• Trainer blueprint: A subclass of BlueprintTrainer, which includes the logic to compute the reward and status of the training, as well as Sensors Actuators.

• Environment definition: A subclass of BlueprintScholaEnvironment, which includes the logic of initializing and resetting the environment between different episodes of training.

• Registering the agent: Connect the agent to the environment definition and trainer.

Initial Setup

1. Create a new blank project with a desired name and location.

2. Install the Schola plugin to the project using the Getting Started with Schola guide.

3. Go to Edit → Project Settings, and scroll down to find Schola.

   Note

   If you don’t see Schola in the Project Settings, please check whether Schola is installed in Edit → Plugins Menu. Please refer to the Getting Started with Schola guide for more information.

   
   

4. For Gym Connector Class, select Python Gym Connector

Creating the Map

1. Create a shooting range with a floor and four walls in the map.

2. For the walls, in Details → Tags, add a new element, and set the value to wall. This tag is used by the RayCastObserver to detect different objects.

Creating the Ball

The Ball class is the projectile that the agent shoots. The ball is spawned by the agent when it takes the shooting action and is destroyed upon hitting a wall or target.

1. Create a new Blueprint Class with parent class Actor, and name it BallShooterBall.

2. Add a Sphere Static Mesh Component to the blueprint, and optionally select a good-looking material.

   1. Enable Details → Physics → Simulate Physics.

   2. Enable Details → Collision → Simulation Generates Hit Events.

   3. Enable Details → Collision → Generate Overlap Events.

   4. Set Details → Collision → Collision Presets to Custom.

   5. Set Details → Collision → Collision Presets → Collision Enabled to Probe Only. This prevents the ball from blocking the agent’s Ray Cast Observer vision.

3. Add a Sphere Collision Component, making it slightly larger than the Sphere.

4. Scale the DefaultSceneRoot to 0.5×0.5×0.5.

Creating the Target

The target is the object that the agent will shoot. The target moves randomly around the map and is destroyed when hit three times by a ball. The Event Tick will apply a random force to the target to move it around the map. The OnTakeAnyDamage_Event will be triggered when hit by a ball, adjust the target’s hitpoint, and destroy the target when the hitpoint reaches zero.

1. Create a new Blueprint Class with parent class Actor, and name it BallShooterTarget.

2. Add a Sphere Static Mesh Component to the blueprint, and optionally select a good-looking material.

   1. Enable Details → Physics → Simulate Physics.

   2. Enable Details → Collision → Simulation Generates Hit Events.

   3. Enable Details → Collision → Generate Overlap Events.

   4. Set Details → Collision → Collision Presets to PhysicsActor.

3. Add a Sphere Collision Component, making it slightly larger than the Sphere.

4. Scale the DefaultSceneRoot to 3x3x3.

5. Add a new boolean variable. Name it isHit. It stores whether the agent is hit by a ball in the current step.

6. Add a new Transform variable. Name it initialTransform. It stores the initial transform of the target when the episode starts.

7. Add a new integer variable. Name it hitPoint, and set the default value to 3. It stores the number of times the target is hit by a ball. The target will be destroyed when the hitpoint reaches zero.

8. Add a new float variable. Name it forceMagnitude, and set the default value to 50. It stores the magnitude of the random force applied to the target on each tick.

9. Create a new function teleportToRandomLocation as shown below, and set the Make Vector node’s random ranges to the range of the shooting range. This function will teleport the target to a random location within the shooting range.

10. Set the Event Graph as shown below.

    1. The Event Begin Play will save the initial transform of the target and bind the OnTakeAnyDamage_Event once.

    2. The OnTakeAnyDamage_Event will be triggered when hit by a ball, adjust the Target’s hitpoint, and destroy the target when the hitpoint reaches zero.

    3. The Event Tick will apply a random force to the target to move it around the shooting range.

11. In Details → Tags, add a new element, and set the value to target. This tag is used by the RayCastObserver to detect different objects.

Creating the Agent

1. Create a new Blueprint Class with parent class Pawn, and name it BallShooterAgent.

2. Add any desired static meshes as the agent’s body, and optionally select good-looking materials.

3. Add an Arrow Component, and set it as the agent’s forward direction. Name it Projectile Indicator.

4. Save and close the blueprint, and place a BallShooterAgent at the center of the map.

Creating the Ball Shooter Shooting Actuator

There are a variety of built-in actuator classes available in Schola, such as the TeleportActuator and MovementInputActuator. However, some games may require custom actuators. In this example, we will create a custom BlueprintDiscreteActuator (subclass of DiscreteActuator) to shoot the ball. This actuator has two possible actions: shoot a ball or do nothing. The GetActionSpace() function will return the action space, and the TakeAction() function will take the action. We will also create two helper functions, getInitialVelocity() and getInitialLocation(), to get the initial velocity and location for spawning the ball.

1. Create a new Blueprint Class with parent class BlueprintDiscreteActuator, and name it BallShooterShootingActuator.

2. Add a new float variable. Name it ballSpawnSpeed, and set the default value to 2000. This stores the speed of the ball when shot.

3. Add a new Rotator variable. Name it projectileSpawnDirection. This stores the direction in which the ball will be spawned. Adjust the values to ensure the ball is spawned in the correct direction.

4. Add a new float variable. Name it ballSpawnOffset. This stores the offset from the agent’s location where the ball will be spawned. Set the default value to 200, and adjust if necessary to ensure the ball is spawned in front of, not inside the agent.

5. Add a new integer variable. Name it countOfBallsShot. It stores the number of balls shot by the agent in the current time step.

6. Add a new Actor variable. Name it Agent. This stores the agent that owns the actuator.

7. Convert the function TakeAction() into an event. This allows us to bind the Ball Hit Event to the spawned ball.

8. Set the getInitialVelocity(), getInitialLocation(), GetActionSpace(), and TakeAction() blueprints as shown below.

Creating the Ball Shooter Discrete Rotation Actuator

Although the RotationActuator exists in Schola and can be used to rotate agents continuously, we will create another custom BlueprintDiscreteActuator (subclass of DiscreteActuator) to rotate the agent. This actuator has three possible actions: rotate left, rotate right, or do nothing.

1. Create a new Blueprint Class with parent class BlueprintDiscreteActuator, and name it BallShooterDiscreteRotationActuator.

2. Add a new float variable. Name it rotationMagnitude, and set the default value to 2. This stores the magnitude of the rotation when the agent rotates.

3. Set the GetActionSpace() and TakeAction() blueprints as shown below.

Creating the Trainer

To train an agent in Schola, the agent must be controlled by an AbstractTrainer, which defines the ComputeReward() and ComputeStatus() functions. In this tutorial, we will be creating an BlueprintTrainer (subclass of AbstractTrainer).

1. Create a new Blueprint Class with parent class BlueprintTrainer, and name it BallShooterTrainer.

2. Add a new integer variable. Name it maxNumberOfHitsPerEpisode. It stores the maximum number of times the agent can hit the target in one episode, which is the number of targets multiplied by the number of hitpoints for each target. It is set by the Environment Definition blueprint.

3. Add a new integer variable. Name it numOfHitsThisEpisode. It stores the number of times the agent has hit the target in the current episode. It is used to determine when the episode ends.

4. Add a new integer variable. Name it numOfTargetHits. It stores the number of times the agent has hit the target in the current step.

5. Add an Actuator component, and set the Details → Actuator Component → Actuator to BallShooterShootingActuator

6. Set the Event Graph as shown below. This binds the On Ball Hit event to any balls spawned by the agent’s actuator, allowing the trainer to detect when the agent hits or misses the target.

Creating the Environment Definition

To train an agent in Schola, the game must have an AbstractScholaEnvironment Unreal object, which contains the agent and logic for initializing or resetting the game environment. In this tutorial, we will be creating an Blueprint Environment (subclass of AbstractScholaEnvironment) as the Environment. The InitializeEnvironment() function is called at the start of the game, and sets the initial state of the environment. In this tutorial, we save the initial transform (position and rotation) of the agent. The ResetEnvironment() function is called before every new episode. In this tutorial, we reset the agent to its initial transform, clean up any leftover balls and targets, spawn three new targets, calculate the TotalHitPoints for the episode, and reset the variables in the trainer.

1. Create a new Blueprint Class with parent class BlueprintScholaEnvironment, and name it BallShooterEnvironment.

2. Add a new variable named agentArray of type Pawn (Object Reference) array. This keeps track of registered agents belonging to this environment definition.

   1. Make this variable publicly editable (by clicking on the eye icon to toggle the visibility).

3. Add a new Transform variable named agentInitialLocation. This is for storing the initial position and rotation of the agent, so it can be restored upon reset.

4. Add a new integer variable named numberOfTargets, and set the default value to 3. This stores the number of targets to spawn in the environment.

5. Add a new integer variable named totalHitPoints. This stores the total number of hit points for the episode, which is the number of targets multiplied by the number of hitpoints for each target.

6. Add a new variable named Targets of type Ball Shooter Target (Object Reference) array. This stores the spawned targets in the environment.

7. Create functions saveAgentInitialTransform and placeAgentToInitialTransform as shown below. This saves the initial transform of the agent and places the agent to its initial transform when the episode starts.

8. Set the Event Graph and RegisterAgents() function as shown below.

9. Save and close the blueprint, and place a BallShooterEnvironment anywhere in the map. The location does not matter.

Registering the Agent

1. Click on the BallShooterEnvironment in the map.

   1. Go to Details panel → Default → Agent Array.

   2. Add a new element.

   3. Select BallShooterAgent in the drop-down menu.

      
      

2. Open the BallShooterAgent class in the blueprint editor.

   1. Go to Details Panel.

   2. Search for AIController.

   3. In the drop-down, select BallShooterTrainer .

      
      

Starting Training

We will train the agent using the Proximal Policy Optimization (PPO) algorithm for 100,000 steps. The following two methods run the same training. Running from the terminal may be more convenient for hyperparameter tuning, while running from the Unreal Editor may be more convenient when editing the game.

schola-sb3 -p 8000 -t 100000 PPO

schola-rllib -p 8000 -t 100000

schola-sb3 -p 8000 -t 100000 --enable-tensorboard --log-dir experiment_ball_shooter PPO

tensorboard --logdir experiment_ball_shooter/PPO_1