Run Optimized LLMs from the Arm AI Portal on Arm Neoverse Cloud Instances

Who is this for?

This Learning Path is for developers and ML engineers running Arm-optimized LLMs on Arm Neoverse Linux machines. It provides a tested ONNX Runtime GenAI workflow and an optional coding-agent workflow for compatible text-to-text packages that use alternative runtimes or formats.

What will you learn?

Upon completion of this Learning Path, you will be able to:

• Prepare an Arm Neoverse Linux machine and download a model from the Arm AI Portal.
• Generate text from the terminal and optionally serve the model through a local web application.
• Explain how the shared application calls the supplied ONNX Runtime GenAI adapter.
• Compare runtime and model-format choices, and optionally use a coding agent to replace the supplied adapter for another compatible text-to-text package.

Prerequisites

Before starting, you will need the following:

• An Arm Neoverse Linux machine running Ubuntu 24.04 LTS, with Python 3.11 or later, for example an AWS m8g.xlarge instance
• At least 16 GB of memory if you plan to use one of the 8B models
• Basic familiarity with Linux command-line tools and Python
• (Optional) Access to a coding agent if you want to generate an adapter for another runtime