Build a Sentiment-Aware Voice Assistant with On-Device LLMs
Introduction
Understand voice sentiment analysis for on-device AI
Set up your environment
Build the voice-to-LLM pipeline
Train the voice sentiment classification model
Convert and quantize the model
Integrate the voice sentiment classification model
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Advanced
Reading time:  1 hr 30 min
Last updated:  24 Aug 2026
Skill level:
  Advanced
Reading time:
  1 hr 30 min
Last updated:
  24 Aug 2026
Author:Bhanu Arya
Arm IP:Cortex-A
Tags: ML Linux Windows macOS Python Transformers ONNX Runtime llama.cpp Gradio
Author:
  • Bhanu Arya
Arm IP:
Cortex-A
Tags:
ML Linux Windows macOS Python Transformers ONNX Runtime llama.cpp Gradio

Who is this for?

This Learning Path is for developers, ML practitioners, and game developers interested in building on-device AI applications, including voice interfaces, real-time interactions with non-player characters (NPCs), and edge AI systems powered by LLMs on Arm platforms.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Build a voice-to-LLM pipeline using Whisper and llama.cpp.
  • Train a voice sentiment classification model using HuBERT on the RAVDESS dataset.
  • Quantize the model and convert into ONNX Runtime for on-device inference.
  • Integrate sentiment classification model with voice-to-LLM pipeline to generate context-aware LLM responses.

Prerequisites

Before starting, you will need the following:

  • Python 3.9 or later for programming.
  • A working microphone for voice input.
  • Basic Python and command-line knowledge.

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll build a sentiment-aware voice assistant that runs on-device on Arm. First, you’ll prepare a UV-managed Python environment, build llama.cpp, and create a Gradio pipeline that transcribes microphone audio with Whisper and sends it to a local LLM. You’ll train a HuBERT classifier on selected RAVDESS sentiments, export and quantize it to ONNX, and add its prediction to the LLM prompt and user interface.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
How do I know the baseline voice-to-LLM pipeline is working?
Record audio and confirm that the interface displays both a transcript and the local LLM response. This verifies that Whisper transcription and the LLM request are working.
Do I need `ffmpeg` installed before using Whisper?
Yes. Install ffmpeg before running the transcription step because Whisper needs it to decode audio.
Where do I save the trained HuBERT model and feature extractor?
After training, save the HuBERT model and its feature extractor in models/hubert_vsa_ravdess. Train it on the selected RAVDESS classes: neutral, happy, and angry.
How do I include the predicted sentiment in the LLM prompt and verify it?
Add the predicted sentiment to the prompt in handle_audio before sending it to the local LLM. To verify the integration, run app.py and confirm that you see a transcript, predicted sentiment, and LLM response.
Which ONNX model does the final voice assistant load?
Set ONNX_PATH to models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx. The final app loads this quantized model with the CPU execution provider.
Next