# Add semantic retrieval and contextual reasoning to Hermes Agent ## In this learning path - [Introduction](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/) - [Explore persistent AI runtime architecture on NVIDIA DGX Spark](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/1_introduce/) - [Build the DGX Spark AI runtime foundation](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/2_build/) - [Deploy Hermes Agent as an orchestration runtime](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/3_deploy_orch_runtime/) - [Add local LLM inference to Hermes Agent](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/4_local_llm/) - [Build persistent semantic memory for Hermes Agent](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/5_persistent_memory/) - [Add semantic retrieval and contextual reasoning to Hermes Agent](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/6_semantic_retrieval/) - [Add autonomous workspace cognition to Hermes Agent](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/7_autonomous_workspace/) - [Next Steps](https://learn.arm.com/learning-paths/laptops-and-desktops/dgx_persistent_agent/_next-steps/) ## Update Hermes to turn memory in Qdrant into a reasoning source In this section, you’ll add semantic retrieval to Hermes Agent. In the previous section, Hermes stored document summaries, embeddings, and metadata in Qdrant. You’ll now turn that stored memory into an active reasoning source: Hermes will accept a question, retrieve relevant memories, assemble them as context, and send that context to the local model. The runtime can already ingest documents, summarize them, generate embeddings, and store semantic memory in Qdrant. You’ll now add a query workflow so Hermes can search memory and use retrieved context to answer questions. The workflow becomes: ``` /workspace/query.txt -> Hermes embeds the question -> Qdrant returns relevant memories -> Hermes assembles context -> Ollama generates a grounded answer ``` This is the first stage where Hermes uses memory as reasoning context instead of only storing it. ### Understand contextual retrieval architecture The retrieval pipeline uses all three runtime services: | Component | Responsibility | |------------------|--------------------------------------------------------------| | Hermes Agent | Detects queries, generates query embeddings, assembles context | | Ollama | Generates embeddings and contextual answers | | Qdrant | Searches stored semantic memory | The runtime keeps the same fixed memory configuration: | Component | Value | |--------------------------|---------------------------------| | Embedding model | `nomic-embed-text` | | Vector dimension | `768` | | Qdrant collection | `workspace_memory` | | Retrieval limit | `3` | Hermes watches for a query file at: ``` /workspace/query.txt ``` When the file exists, Hermes reads the question, deletes the query file, searches memory, and prints the answer in the container logs. ### Add retrieval functions to Hermes Open and edit the file `~/dgx-hermes-agent/hermes/agent.py`. Replace the file with the following version: ```python import os import uuid import time import ollama from qdrant_client import QdrantClient from qdrant_client.models import ( Distance, VectorParams, PointStruct ) from watchdog.observers import Observer from watchdog.events import FileSystemEventHandler WATCH_DIR = "/workspace/inbox" SUPPORTED_EXTENSIONS = [ ".txt", ".md", ".log" ] OLLAMA_HOST = os.getenv( "OLLAMA_HOST", "http://ollama:11434" ) QDRANT_HOST = os.getenv( "QDRANT_HOST", "qdrant" ) COLLECTION_NAME = "workspace_memory" client = ollama.Client(host=OLLAMA_HOST) qdrant = QdrantClient( host=QDRANT_HOST, port=6333 ) def ensure_collection(): collections = qdrant.get_collections().collections names = [c.name for c in collections] if COLLECTION_NAME not in names: qdrant.create_collection( collection_name=COLLECTION_NAME, vectors_config=VectorParams( size=768, distance=Distance.COSINE ) ) print(f"[Memory] Created collection: {COLLECTION_NAME}") class WorkspaceHandler(FileSystemEventHandler): def on_created(self, event): if event.is_directory: return filename = os.path.basename(event.src_path) if filename.startswith("."): return ext = os.path.splitext(filename)[1] if ext not in SUPPORTED_EXTENSIONS: return print(f"\n[Agent] New file detected:") print(event.src_path) process_file(event.src_path) def generate_summary(content): response = client.chat( model="qwen2.5:7b", messages=[ { "role": "system", "content": ( "You are a local AI workspace assistant. " "Summarize the document in 3 concise bullet points." ) }, { "role": "user", "content": content[:4000] } ] ) return response["message"]["content"] def generate_embedding(content): response = client.embed( model="nomic-embed-text", input=content[:4000] ) return response["embeddings"][0] def store_memory(path, content, summary, embedding): point_id = str(uuid.uuid4()) qdrant.upsert( collection_name=COLLECTION_NAME, points=[ PointStruct( id=point_id, vector=embedding, payload={ "path": path, "summary": summary, "content": content[:4000] } ) ] ) print(f"[Memory] Stored document: {path}") def search_memory(query): print("\n[Memory] Searching semantic memory...") embedding = generate_embedding(query) results = qdrant.query_points( collection_name=COLLECTION_NAME, query=embedding, limit=3 ).points memories = [] for result in results: payload = result.payload memories.append({ "path": payload.get("path"), "summary": payload.get("summary"), }) return memories def query_workspace(question): memories = search_memory(question) context = "\n\n".join([ f"Document: {m['path']}\nSummary:\n{m['summary']}" for m in memories ]) response = client.chat( model="qwen2.5:7b", messages=[ { "role": "system", "content": ( "You are a persistent AI workspace assistant. " "Answer questions using the retrieved workspace memory." ) }, { "role": "user", "content": ( f"Question:\n{question}\n\n" f"Relevant workspace memory:\n{context}" ) }, ] ) answer = response["message"]["content"] print("\n[Workspace Query]") print(question) print("\n[Retrieved Memories]") print(context) print("\n[AI Response]") print(answer) def process_file(path): try: with open(path, "r", encoding="utf-8") as f: content = f.read() print("\n[Agent] Running summarization inference...") summary = generate_summary(content) print("\n[Agent] AI Summary:") print(summary) print("\n[Agent] Generating embeddings...") embedding = generate_embedding(content) store_memory( path, content, summary, embedding ) except Exception as e: print(f"[Agent] Error: {e}") if __name__ == "__main__": print("\n[Hermes Agent] Starting workspace watcher...") print(f"[Hermes Agent] Monitoring: {WATCH_DIR}") ensure_collection() observer = Observer() observer.schedule( WorkspaceHandler(), WATCH_DIR, recursive=False ) observer.start() try: while True: time.sleep(1) if os.path.exists("/workspace/query.txt"): with open("/workspace/query.txt", "r") as f: question = f.read().strip() os.remove("/workspace/query.txt") query_workspace(question) except KeyboardInterrupt: observer.stop() observer.join() ``` ### Understand code trace `search_memory()` embeds the user question with `generate_embedding()`, then calls `qdrant.query_points()` with `query=embedding` to retrieve the three closest semantic matches. Each result’s payload is read from `result.payload`, and the path and summary are assembled into a list of memories. `query_workspace()` joins those memories into a context string, then sends both the original question and the assembled context to `qwen2.5:7b` using the chat API. The model receives prior workspace memory as part of the prompt, so its answer is grounded in ingested documents rather than general knowledge. The main runtime loop checks for `/workspace/query.txt` on each iteration. When the file exists, Hermes reads the question, deletes the file immediately to avoid re-processing, and calls `query_workspace()`. ### Understand runtime compatibility considerations Use `qdrant.query_points()` with `query=embedding` when calling the vector search API. Older examples use `query_vector=embedding`, which isn’t compatible with current versions of the Qdrant Python client. Read each result’s payload with `result.payload` before assembling the retrieval context. The `limit=3` value in `search_memory()` is hardcoded to keep the retrieval behavior easy to inspect. You can make it configurable later. ### Rebuild Hermes with retrieval functions Rebuild the Hermes container: ``` cd ~/dgx-hermes-agent/compose docker compose build hermes ``` Restart the runtime: ``` docker compose up -d ``` Follow the Hermes logs: ``` docker logs -f hermes ``` Leave this terminal open with the log stream running and open a second terminal for the next step. The startup output is similar to: ``` __output__[Hermes Agent] Starting workspace watcher... __output__[Hermes Agent] Monitoring: /workspace/inbox ``` ## Validate semantic retrieval and contextual reasoning After adding semantic retrieval and contextual reasoning to Hermes, verify that it works as expected. ### Create test memory In a second terminal, create a few new documents so Qdrant contains useful semantic memory. For each document, write the file in `/tmp` first and then move it into `workspace/inbox/`. This gives Hermes a completed file when the `on_created()` event fires. Create a document about CPU orchestration: ``` cat > /tmp/cpu-orchestration-note.txt << 'EOF' Arm CPUs are responsible for orchestration in persistent AI runtimes. They coordinate filesystem events, runtime scheduling, container services, document parsing, metadata handling, and vector database operations. EOF mv /tmp/cpu-orchestration-note.txt \ ~/dgx-hermes-agent/workspace/inbox/cpu-orchestration-note.txt ``` Create a document about GPU inference: ``` cat > /tmp/gpu-inference-note.txt << 'EOF' NVIDIA GPUs accelerate local model inference, token generation, summarization, embedding generation, and contextual reasoning workloads. EOF mv /tmp/gpu-inference-note.txt \ ~/dgx-hermes-agent/workspace/inbox/gpu-inference-note.txt ``` Create a document about semantic memory: ``` cat > /tmp/semantic-memory-note.txt << 'EOF' Semantic memory stores embeddings and metadata in a vector database. This allows persistent AI systems to retrieve relevant prior context based on meaning instead of exact keyword matching. EOF mv /tmp/semantic-memory-note.txt \ ~/dgx-hermes-agent/workspace/inbox/semantic-memory-note.txt ``` Watch the first terminal running Hermes logs until each document is summarized, embedded, and stored. The output is similar to: ``` __output__[Agent] New file detected: /workspace/inbox/cpu-orchestration-note.txt __output__ [Agent] Running summarization inference... __output__ [Agent] AI Summary: - Arm CPUs handle orchestration in persistent AI runtimes. - They manage filesystem events, runtime scheduling, and container services. - Tasks also include document parsing, metadata handling, and vector database operations. __output__ [Agent] Generating embeddings... __output__[Memory] Stored document: /workspace/inbox/cpu-orchestration-note.txt ``` ### Test semantic retrieval In the second terminal, create a query file: ``` echo "How do CPUs help persistent AI systems?" \ > ~/dgx-hermes-agent/workspace/query.txt ``` Hermes checks for `/workspace/query.txt` in the runtime loop. When it sees the file, it reads the question and removes the file. It then embeds the question, searches Qdrant, and sends the retrieved context to Ollama. In the first terminal running Hermes logs, confirm that semantic search started. The output is similar to: ``` __output__[Memory] Searching semantic memory... ``` Then confirm that Hermes printed the question and the retrieved memory context. The output is similar to: ``` __output__[Workspace Query] __output__How do CPUs help persistent AI systems? __output__ __output__[Retrieved Memories] __output__Document: /workspace/inbox/cpu-orchestration-note.txt __output__Summary: __output__- Arm CPUs manage orchestration in persistent AI runtimes. __output__- They handle filesystem events, runtime scheduling, and container services. __output__- Additionally, they process document parsing, metadata handling, and vector database operations. __output__ __output__Document: /workspace/inbox/cpu-orchestration-note.txt __output__Summary: __output__- Arm CPUs handle orchestration in persistent AI runtimes. __output__- They manage filesystem events, runtime scheduling, and container services. __output__- Tasks also include document parsing, metadata handling, and vector database operations. __output__ __output__Document: /workspace/inbox/semantic-memory-note.txt __output__Summary: __output__- Semantic memory uses a vector database to store embeddings and metadata. __output__- This enables persistent AI systems to recall relevant past context based on meaning. __output__- Retrieval is done without relying solely on exact keyword matching. ``` Finally, confirm that Ollama generated a response from the retrieved context: ``` __output__[AI Response] __output__In persistent AI systems, CPUs play a crucial role in managing orchestration tasks that are essential for the system's operation. Specifically, Arm CPUs handle several key responsibilities: ``` ### Verify contextual reasoning In the second terminal, ask another question: ``` echo "Why does the runtime need semantic memory?" \ > ~/dgx-hermes-agent/workspace/query.txt ``` Hermes embeds the question and Qdrant retrieves relevant summaries. Hermes then assembles those summaries into context, and Ollama generates an answer grounded in the retrieved memory. The logs include a retrieved memory from the semantic memory document that you created earlier. The output is similar to: ``` __output__[Memory] Searching semantic memory... __output__ __output__[Workspace Query] __output__Why does the runtime need semantic memory? __output__ __output__[Retrieved Memories] __output__Document: /workspace/inbox/memory-test.txt __output__Summary: __output__- Persistent AI runtimes require memory to incorporate past workspace activities into future reasoning. __output__- Semantic memory in AI systems retains embeddings and metadata to store relevant context. __output__- This stored information allows for retrieval of pertinent context, enhancing the runtime's ability to reason effectively. ``` Finally, confirm that Ollama generated a response from the retrieved context: ``` __output__[AI Response] __output__The runtime needs semantic memory because it retains embeddings and metadata that store relevant context from past workspace activities. This allows the system to effectively reason by retrieving pertinent information, enhancing its ability to understand and respond to new inputs more intelligently. Unlike simple keyword matching, semantic memory uses a vector database which captures the meaning of words or concepts, enabling more accurate and contextualized recall of past events or knowledge. ``` ## Understand CPU and GPU responsibilities The Arm Grace CPU coordinates the retrieval workflow: watching for `query.txt`, reading and deleting the query file, calling Ollama for the query embedding, searching Qdrant for relevant memories, parsing result payloads, assembling the retrieved context, and calling Ollama again for contextual reasoning. The Blackwell GPU accelerates query embedding generation, contextual LLM inference, and response generation. Qdrant performs the vector similarity search and returns the most relevant memory payloads. This retrieval workflow creates a local contextual reasoning loop using persistent memory. ## What you’ve accomplished and what’s next You’ve added semantic retrieval and contextual reasoning to Hermes Agent. The runtime now turns a question into an embedding, searches Qdrant with `query_points(...)`, assembles retrieved memory, and sends that context to `qwen2.5:7b`. The runtime can now store memory and reason over it through the local `/workspace/query.txt` workflow. Next, you’ll add autonomous workspace cognition. Before moving to the next section, press `Ctrl+C` in the first terminal running Hermes logs to stop the Hermes log stream. In the next section, you’ll rebuild the Hermes container and run `docker logs -f hermes` again.