Download and run a model

Set the model ID

Choose the text-to-text model you want to use from the Arm AI Portal. The Hugging Face repository ID for each model contains the organization and repository name, for example Arm/<model-repository-name>. The included runner scripts have been confirmed with the models in the hidden table below. Use the table below to copy the ID.

Set MODEL_ID to that value. Replace the placeholder before running the command:

    

        
        
export MODEL_ID="Arm/<model-repository-name>"

    

The remaining commands use MODEL_ID, so you only need to substitute the repository ID once. Keep the same terminal session, or export the variable again if you open a new one.

For information on how each script works, see Understand the runner scripts .

Download the model

Activate the Python environment and download the selected model:

    

        
        
source .venv/bin/activate
python download_model.py --repo-id "$MODEL_ID"

    

If the optional adapter workflow already downloaded the package, this command reuses the local snapshot.

The script stores the complete model package under models/ and replaces the organization/repository separator / with __, so each Hugging Face repository ID maps to one predictable directory name. The supplied model_adapter.py validates the ONNX Runtime GenAI files before loading the model.

Validate the model package against the supplied adapter:

    

        
        
python validate_adapter.py --model-id "$MODEL_ID"

    

The supported models use the same ONNX Runtime GenAI execution path, but they expect different prompt formats. Instruction-tuned models use the chat or instruction template packaged for the model. Base models usually receive plain text to continue without a chat template.

For an instruction-tuned model, the output is similar to:

    

        
        Adapter validation passed
Runtime: ONNX Runtime GenAI
Interaction mode: chat
Model directory contains the files required by the adapter

        
    

A base model reports completion as its interaction mode.

A different generated adapter would report its selected runtime and either chat or completion according to the model package it supports.

Run text generation

Choose the tab that matches the model type shown in the Arm AI Portal or model card. The supplied ONNX adapter automatically selects raw completion text, the chat template packaged with the model, or the built-in Qwen2.5 ChatML fallback, depending on the model provided. A generated adapter will use the prompt handling implemented for its selected package.

    

        
        

python run_model.py \
  --model-id "$MODEL_ID" \
  --prompt 'Write a concise technical briefing for a cloud architect explaining why Arm64 CPUs are well suited to AI inference. Cover performance per watt, deployment scale, and software portability.'
  

    

    

        
        

python run_model.py \
  --model-id "$MODEL_ID" \
  --max-new-tokens 64 \
  --prompt 'Title: Why Arm64 is reshaping cloud infrastructure

Arm64 is becoming a preferred architecture for modern cloud workloads because'
  

    

Instruction-tuned models apply the chat template included with the model bundle. Base models receive the article title and opening sentence as raw text to continue.

The model output varies, but the final lines contain generation statistics similar to:

    

        
        Generated tokens: 128
Time to first token: 0.74 seconds
Decode throughput: 42.18 tokens/second

        
    

These values are illustrative. They depend on the model, processor, thread count, prompt length, and system load.

For details about base-model prompts, chat templates, the Qwen2.5 fallback, and Qwen3 thinking mode, see Understand the runner scripts .

(Optional) Use the web interface

You can optionally serve the models through a simple browser interface.

    

        
        
ssh -L 8000:127.0.0.1:8000 user@remotehost

    

On your original terminal, with the venv activated, and in the arm-llm-cloud directory, install the web dependencies on the Neoverse machine:

    

        
        
python -m pip install "fastapi>=0.116,<1" "uvicorn>=0.35,<1"

    

Choose the tab that matches your model type. Start the server, leave it running, and open the webpage in your local browser.

    

        
        
python genai_web.py --model-id "$MODEL_ID"

    

    

        
        
python genai_web.py \
  --model-id "$MODEL_ID" \
  --max-new-tokens 64

    

What you’ve accomplished

You have downloaded an ONNX Runtime GenAI model and generated text through the shared terminal application. If you completed the optional section, you also used the same adapter through a browser interface. Next, you will inspect the boundary between the shared applications and the supplied ONNX adapter.