How an AI Agent Reads Your Memory on the Next Turn

The cleanest mental model for agent memory is one sentence: the agent reads memory from disk once, at the start of each new session, and then never reads it again. Everything the agent “remembers” mid-session comes from conversation history; everything it remembers across sessions comes from the memory file. If you only carry one model into your operator practice, carry that one.

This guide walks the full read path — disk to compiled prompt to next assistant turn — and the discipline that separates well-curated memory from a junk drawer that quietly biases every future reply.

What “memory” actually is

Memory, in Hermes Agent, is a small file (~/.hermes/memory.md, ~/.hermes/user.md, or operator-specific) that the system reads at the start of every new session and inserts verbatim into the system prompt. The agent does not load additional memory mid-session unless the operator explicitly asks for a specific memory file to be re-loaded.

The memory is then “in” the conversation. Every turn, the agent sees its content. The agent cannot forget it; it is part of the system prompt until the session ends.

The memory read path

$ ls -la ~/.hermes/memory.md
-rw-r--r-- 1 user user 1.2K ... ~/.hermes/memory.md

The full sequence at session start:

1. The Hermes runtime loads ~/.hermes/config.yaml to find the operator’s session profile.
2. The profile lists which memory files to read. Default: memory.md, user.md (Alastair’s setup).
3. The runtime reads those files from disk.
4. The contents are concatenated and inserted into the system prompt as always-loaded context.
5. The agent sees the content on every turn until the session ends.

Across sessions, the file content is durable — edits to the file on disk are picked up by the NEXT session, not the current one.

Memory vs conversation history

Two kinds of “memory” of the agent:

• Conversation history is in-memory and survives only as long as the session is alive. When the session ends, the conversation history evaporates (unless explicitly saved).
• Memory is on-disk and survives across sessions.

A common mistake: assuming the agent “remembers” something from a previous session because of conversation history. It doesn’t — the conversation is gone. Memory is the only thing that carries forward.

How memory enters the prompt

The system prompt at every turn reads:

[system prompt]
   ↓
[memory: ~/.hermes/memory.md, ~/.hermes/user.md]
   ↓
[tools]
   ↓
[conversation history]
   ↓
[user's current message]

This is the canonical compiled-prompt order. Note that memory comes before tools and before conversation history. The agent sees memory before it sees anything else; memory informs every assistant turn.

What memory is NOT

Three confusions:

1. Memory is not skills. Skills are on-demand (skill_view); memory is always-loaded. Skills ride the conversation layer; memory rides the system-prompt layer.
2. Memory is not user settings. Hermes config (~/.hermes/config.yaml) is operator-side; memory is operator-side too, but content-only — it’s text, not config.
3. Memory is not chat history. Even an infinite chat history won’t survive across sessions; only memory does.

The discipline of well-curated memory

The Johnny5 routing gate (memory additions go through a memory-quality check) exists because memory that’s noisy is worse than memory that’s clean:

• One fact per entry. Don’t pack three preferences into one entry.
• Stable facts only. “Be concise today” is a per-session preference; “Be concise” is a durable preference; “User is married, has a Mac mini, lives in Edinburgh” is identity-grade.
• Cross-cutting facts only. Per-project rules belong in project context, not memory. Memory is for facts that are true in any project.
• No internal commentary. Memory is for facts, not opinions. The agent doesn’t need to know why you like haiku; it needs to know you pin haiku.

What the agent does NOT do with memory

1. Memory is not patched mid-session. The agent doesn’t “update” memory by writing a new entry; that’s an operator action via memory tool or ~/.hermes/memory.md edit.
2. Memory is not auto-cleaned. Stale entries stay until the operator removes them.
3. Memory is not summarized by the agent. If an entry is too verbose, the agent sees all of it; trim on the operator side.

Why this matters in practice

The asymmetry between memory and conversation history produces a specific failure mode: the agent gives confident, context-aware replies in long sessions and the operator starts to assume the agent “knows” them. Then a new session starts, the conversation is gone, and the agent reverts to a stranger — because the only durable signal was the memory file, which was small or stale or both.

Two practical implications:

• End-of-session review is the high-leverage moment. Right before a session closes, scan the conversation for facts that should graduate to memory: durable preferences, identity-grade details, recurring project conventions. Add them once, on disk, not as a paragraph in chat.
• Treat memory as small on purpose. A 200-byte memory file outperforms a 4KB one, because every byte competes with system prompt, tools, and history for context window. The Johnny5 routing gate keeps entries atomic for the same reason — small facts survive compaction; sprawling ones get trimmed or ignored.

The verification checklist below is the smallest proof that the model is correct on your machine, not just in the docs.

Verification checklist

#	Question	What it proves
1	Open ~/.hermes/memory.md — is the content short?	Should be < 1KB in most cases
2	Run hermes chat -q "What time is it". Check the response cite.	Memory loaded?
3	Edit memory on disk; start a new session; check the response	Edits took effect
4	Save a fact in this session; check that it’s NOT in next session’s memory	Only persisted = memory

Sources

• Hermes Agent memory docs
• ABS companion: Hermes Agent Memory vs Skills
• ABS companion: Hermes Protocol: The System Prompt, the Context Window, and the Rule Boundary

Last verified: 2026-07-14 against Hermes v0.18.2 (2026.7.7.2).