Train multi-agent reinforcement learning policies with MAPPO on an Arm-based cloud instance
Introduction
Prepare the Arm cloud environment
Configure and train MAPPO
Locate and validate the training checkpoint
(Optional) Deploy the checkpoint to a cloud GUI
Export the MAPPO actor for inference
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Advanced
Reading time:  5 hrs 30 min
Last updated:  03 Sep 2026
Skill level:
  Advanced
Reading time:
  5 hrs 30 min
Last updated:
  03 Sep 2026
Authors:Sagar Surendran, Arm
Na Li, Arm
Masoud Koleini
Arm IP:Neoverse
Tags: ML Linux Reinforcement Learning Multi-Agent Reinforcement Learning MAPPO Python PyTorch TorchRL BenchMARL VMAS
Authors:
  • Sagar Surendran, Arm
  • Na Li, Arm
  • Masoud Koleini
Arm IP:
Neoverse
Tags:
ML Linux Reinforcement Learning Multi-Agent Reinforcement Learning MAPPO Python PyTorch TorchRL BenchMARL VMAS

Who is this for?

This Learning Path is for cloud and machine learning developers who want to train and package multi-agent reinforcement learning navigation policies on Arm-based servers.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Configure a vectorized BenchMARL and VMAS workload for an Arm cloud instance
  • Train and evaluate a multi-agent navigation policy with MAPPO
  • Package and validate the trained BenchMARL checkpoint when the companion cloud visualization GUI is available
  • Extract and validate the shared actor as a smaller inference-only artifact

Prerequisites

Before starting, you will need the following:

  • An aarch64 cloud instance running Ubuntu 24.04 with SSH access, sudo privileges, and internet access
  • Familiarity with Linux, Python, PyTorch, and reinforcement learning concepts such as observations, actions, rewards, and policies
  • A local checkout of the companion MARL GUI containing tools/deploy_checkpoint.py and tools/inspect_checkpoint.py if you want to deploy the BenchMARL checkpoint to a GUI

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll configure and run a vectorized MAPPO navigation workload on an Arm-based cloud instance with BenchMARL, VMAS, and TorchRL. First, you’ll select the agent count and CPU devices for shared-actor training with a centralized critic. Then, you’ll restore the run configuration, locate and validate the checkpoint, optionally deploy it to a companion GUI, and export the actor for inference.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
How many agents should I use for the reference MAPPO run?
Use three agents by exporting AGENTS=3 before configuring the run. With share_policy_params=true, the three agents share one actor while receiving different observations.
How do I restore a previous run’s configuration before validating the checkpoint?
Activate the training environment, set RUN_DIR to the run directory that you want to validate, and source "$RUN_DIR/run.env". Then, change to the BenchMARL repository to continue validation.
How do I make sure sampling and training run on the CPU?
Export SAMPLING_DEVICE=cpu and TRAIN_DEVICE=cpu before you launch the experiment. BenchMARL configures sampling and training devices independently, and saves these choices in your run environment.
What should I expect from the exported actor artifact?
Your export contains only the actor policy, without the critic, replay buffer, collector state, or training counters. For VMAS navigation, you’ll get an MLP with 18 inputs, two 256-unit Tanh layers, and four outputs for a two-dimensional action.
Do I need the GUI to verify the checkpoint or export the actor?
No. You can validate the checkpoint and export the actor without the GUI. Use the GUI only when you have the companion checkout described in the prerequisites. Otherwise, skip that stage and continue with exporting the actor.
Next