Build an offline voice chatbot with faster-whisper and vLLM on DGX Spark
Introduction
Build an offline voice assistant with whisper and vLLM
Install faster-whisper for local speech recognition
Build a real-time STT pipeline on CPU
Fine-tune segmentation parameters
Build a real-time offline voice chatbot using STT and vLLM
Connect speech recognition to vLLM for real-time voice interaction
Specialize offline voice assistants for customer service
Enable context-aware dialogue with short-term memory
Next Steps

  Log an issue

  Fork and edit

  Discuss on Discord

About this Learning Path

Skill level:  Advanced
Reading time:  1 hr
Last updated:  29 Jul 2026
Skill level:
  Advanced
Reading time:
  1 hr
Last updated:
  29 Jul 2026
Author:Odin Shen, Arm
Arm IP:Neoverse
Tags: Performance and Architecture Linux Docker Python
Author:
  • Odin Shen, Arm
Arm IP:
Neoverse
Tags:
Performance and Architecture Linux Docker Python

Who is this for?

This is an advanced topic for developers and ML engineers who want to build private, offline voice assistant systems on Arm-based servers such as DGX Spark.

What will you learn?

Upon completion of this Learning Path, you will be able to:

  • Explain the architecture of an offline voice chatbot pipeline combining speech-to-text (STT) and vLLM
  • Capture and segment real-time audio using PyAudio and Voice Activity Detection (VAD)
  • Transcribe speech using faster-whisper and generate replies using vLLM
  • Tune segmentation and prompt strategies to improve latency and response quality
  • Deploy and run the full pipeline on Arm-based systems such as DGX Spark

Prerequisites

Before starting, you will need the following:

  • An NVIDIA DGX Spark system with at least 15 GB of available disk space
  • A USB microphone for audio input

Summary

AI-assisted

This summary was drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
You’ll build a local voice chatbot on an Arm-based NVIDIA DGX Spark with faster-whisper for speech-to-text and vLLM for response generation. You’ll capture microphone audio with PyAudio, add voice and turn detection, tune segmentation for stable low-latency transcription, and connect the CPU STT pipeline to GPU-backed vLLM. You’ll validate segmented transcripts followed by local replies.

Frequently asked questions

AI-assisted

These FAQs were drafted with an approved AI-assisted workflow and reviewed by Arm contributors before publication. Human technical review remains part of the process so the final page reflects engineering rigor, accuracy, and Arm editorial standards.

Close
?
What result should I expect after installing `faster-whisper`?
After a successful installation, you can transcribe a short audio sample or live microphone input with readable text and no runtime errors. Use this to confirm the installation before moving on to pipeline changes.
When should I upgrade the speech model in the CPU STT pipeline?
Upgrade after you confirm baseline transcription works. The build step adds a more accurate model and VAD. If latency increases, proceed to segmentation tuning.
How do I know VAD and turn detection are working correctly?
Transcriptions should arrive as sentence-like chunks, and pauses should start new segments. If long monologues merge into one block or speech is cut mid-sentence, adjust the segmentation parameters.
What should I verify before integrating vLLM with the STT engine?
Ensure the CPU-based STT runs in real time on your DGX Spark and produces stable, segmented text. A clean, timely text stream simplifies downstream integration with vLLM.
What behavior confirms the end-to-end offline chatbot is running?
Speak into the microphone and watch for segmented transcriptions from faster-whisper, followed by a locally generated reply from vLLM. Seeing this sequence consistently indicates the pipeline is integrated and running on the system.
Next