Run Optimized LLMs from the Arm AI Portal on Arm Neoverse Cloud Instances
Introduction
Prepare the Arm Neoverse Linux environment
Download and run a model
Understand the runner scripts
Explore other cloud LLM deployment options
Next steps
Run Optimized LLMs from the Arm AI Portal on Arm Neoverse Cloud Instances
You have run an Arm AI Portal model using ONNX Runtime GenAI and inspected the scripts behind the workflow. That stack is one way to deploy text generation on an Arm cloud machine, but another model format, serving requirement, or operational constraint might lead you to a different runtime.
The supplied workflow uses ONNX Runtime GenAI, but the same deployment components apply to packages that use other model formats and runtimes. Use this page to assess which application components need to change for an alternative package.
A cloud text-generation application needs more than model weights. The model format must match the inference runtime, the tokenizer must match the model, and the host must provide enough memory and supported CPU instructions. You also need a way to submit prompts and return generated tokens.
Identify the components you might replace
The ONNX Runtime GenAI example supplies one choice for each component in the deployment. If you adapt the application, identify which parts can stay and which parts need to change:
| Component | Purpose | Where to get it |
|---|---|---|
| Model artifact | Contains the trained weights in a runtime-compatible format such as ONNX, GGUF, or Safetensors | Select and download the model from the Arm AI Portal |
| Model configuration | Describes the model architecture, tensor names, generation settings, and special tokens | Included with the selected Arm AI Portal model package |
| Tokenizer and prompt template | Converts text to token IDs and formats prompts for instruction-tuned or base models | Included with the selected Arm AI Portal model package |
| Inference runtime | Loads the model, manages token generation, and selects the CPU execution path | The runtime project’s package registry, container registry, or source repository |
| Serving interface | Exposes the model through a command line, web application, or HTTP API | Included with the runtime or added by your application |
| Arm compute | Supplies CPU cores, memory, storage, and network access | An Arm-based cloud VM or Arm Neoverse server |
| Operational controls | Add authentication, health checks, logging, metrics, and request limits | Your application and cloud platform services |
The selected Arm AI Portal package supplies the model artifact, configuration, tokenizer, and prompt template. Keep these files together because a tokenizer or chat template from a different model revision can generate incorrect token IDs or prompt formatting even when the runtime loads the weights successfully.
Compare alternative self-managed runtimes
The inference runtime is the component most likely to change when you select another model format. Each runtime expects a particular artifact format and package layout.
| Runtime | Compatible Arm AI Portal package | Where to get the runtime | Common use |
|---|---|---|---|
| ONNX Runtime GenAI | ONNX graph, genai_config.json, tokenizer files, and an optional chat template | ONNX Runtime GenAI installation documentation | Quantized CPU inference with explicit control over prompt preparation and token generation |
llama.cpp | A GGUF model and tokenizer metadata, when offered for the selected model | llama.cpp source, binaries, and server
| Lightweight local or cloud CPU inference with a command line or OpenAI-compatible server |
| vLLM | A model package supported by vLLM, when offered for the selected model | vLLM CPU installation documentation | Throughput-oriented serving and concurrent API requests |
| Transformers | Model weights, configuration, tokenizer, and optional chat template, when offered for the selected model | Transformers installation documentation | Prototyping, compatibility checks, and direct Python inference |
Artifact formats are not interchangeable. For example, an ONNX Runtime GenAI application cannot load a GGUF file directly, and llama.cpp does not use an ONNX model bundle. In the Arm AI Portal, select a package prepared for the runtime you plan to use.
Compare Arm CPU optimization support
The previous page explains how the supplied ONNX Runtime path can select KleidiAI kernels automatically for compatible operations. Other runtimes handle KleidiAI differently:
- For
llama.cpp, build withGGML_CPU_KLEIDIAI=ONto include the KleidiAI CPU backend. The runtime then selects applicable kernels based on detected CPU features . - For PyTorch, supported low-bit TorchAO operators can use dynamic kernel selection with KleidiAI .
Runtime support alone does not guarantee KleidiAI dispatch. The runtime build, model operators, quantization, tensor shapes, and processor features must also be compatible. Check the selected runtime and package before making performance assumptions.
What you’ve learned
You can now compare the completed ONNX Runtime GenAI workflow with other self-managed runtimes and identify which model, runtime, serving, or operational component needs to change for another cloud LLM deployment.