Run a local AI agent with Ollama to visualize CPU orchestration on Arm
Introduction
Understand the agent and how work is split between CPU and GPU
Set up your environment before running the agent
Serve a model locally with Ollama
Understand the concierge agent code
Run the agent and read the timeline
Understand the CPU's role in the agentic workflow
Next Steps
Run a local AI agent with Ollama to visualize CPU orchestration on Arm
Run AI models locally with Ollama
The agent’s reasoning steps, such as choosing search terms, selecting URLs, and writing the final summary, are handled by an LLM. Instead of calling a cloud API, you’ll serve the model locally with Ollama .
Ollama is a lightweight runtime that downloads open models, loads them into memory, and exposes a local HTTP API at http://localhost:11434. When the agent calls that endpoint, the model runs directly on your machine: the CPU and GPU on your MacBook, Arm Linux laptop, or NVIDIA DGX Spark.
Running locally has three benefits that matter for an agent:
- Privacy: your prompts and the web content the agent reads stay on the device.
- Cost: there are no per-token API charges, so you can run many queries freely.
- Control: you choose the exact model and keep it resident in memory for fast repeated calls.
Install and start Ollama
Install
Ollama
and start its server. The server exposes a local API at http://localhost:11434 that the agent connects to.
- Install Ollama using one of the following options:
brew install ollama
curl -fsSL https://ollama.com/install.sh | sh
- Start the Ollama server using the same method you used to install it:
brew services start ollama
ollama serve
With
Homebrew
, brew services runs Ollama in the background, so you can use the same terminal throughout. To stop it later, run brew services stop ollama.
With the install script, ollama serve runs in the foreground. Keep that terminal open and use a second terminal for the remaining commands. To stop it later, press Ctrl+C in the terminal running ollama serve.
Pull the Gemma model
You’ll use
Gemma 3
, Google’s family of open models. The 4-billion-parameter size (gemma3:4b) is a good default: it’s capable enough for the agent’s reasoning steps and small enough to run on a laptop.
Download the model:
ollama pull gemma3:4b
Confirm the model is available:
ollama list
You’ll see gemma3:4b in the list of installed models.
(Optional) Choose a different model
The agent reads the model name from the OLLAMA_MODEL environment variable, so you can switch models without editing the code. The default in the script is gemma3:4b.
| Model | Size | Best for |
|---|---|---|
gemma3:4b | 4B | Laptops and modest hardware; the default |
gemma3:27b | 27B | High-memory systems such as DGX Spark, for stronger reasoning |
To use a larger model on a DGX Spark, pull it and set the environment variable before running the agent.
For example, to use a model with 27 billion parameters:
ollama pull gemma3:27b
export OLLAMA_MODEL="gemma3:27b"
Larger models produce stronger summaries but use more memory and generate tokens more slowly. Start with gemma3:4b to confirm everything works, then experiment with larger models if your hardware allows.
Test the model
Before wiring it into the agent, confirm the model responds:
ollama run gemma3:4b
Enter a short prompt:
Summarize what a local AI agent does in one sentence.
Type /bye to exit the model session.
What you’ve accomplished and what’s next
You’ve now set up Ollama, served a Google gemma3:4b model locally, and tested the model.
Next, you’ll look at how the agent code uses the model.