How to find, download, and evaluate local models on Hermes Agent or OpenClaw

The model with the best leaderboard score is not necessarily the best one for your machine or your work. This playbook gives you a shorter loop: shortlist models that fit, download one with Ollama, connect it to your agent, then test it on prompts you actually use.

The shared work comes first. After that, follow only the Hermes Agent or OpenClaw section. You do not need to read both.

Before you start

You need:

• Ollama running locally. curl -fsS http://localhost:11434/v1/models should return JSON.
• Roughly 10 GB of free disk for an initial 7B–8B model.
• Three to five real tasks, saved as prompts or listed in one text file.
• Hermes Agent or OpenClaw installed and reachable from the same machine.

If Ollama is not installed, use Local LLM with Ollama: The Easy Mode first.

The procedure at a glance

Step	Checkpoint	Roll back when it fails
1. Find and download	Ollama lists the exact model ID	Pick a smaller model, correct the tag, or free disk
2. Connect your agent	A one-line prompt returns through the local model	Recheck the server, base URL, and exact model ID
3. Evaluate and decide	Every candidate ran the same tasks and has a recorded score	Fix the task set or server error before comparing models

Step 1 — Find and download a model

Start at the Ollama library. Build a shortlist of three candidates, then judge each on five things:

Question	Practical rule
Does it fit?	Start with 7B–8B on a 16 GB machine. Leave memory headroom for the OS, context, and agent tools.
Is the license usable?	Read the model card if outputs will be used commercially or redistributed.
Is the context long enough?	Hermes requires at least 64,000 tokens. OpenClaw also warns against small contexts for full agent runs.
Does evidence support it?	Treat public benchmarks as a filter, not the final decision.
Is it tuned for your task?	Prefer a coding model for coding and a general model for chat.

Do not pull five giant models “just to see.” Pull the smallest plausible candidate first:

ollama pull llama3.1:8b

Command map

Command	Job	Observable success
ollama list	Check models already on disk	A table of model IDs and sizes
ollama pull <model>:<tag>	Download one candidate	The command finishes successfully
curl -fsS http://localhost:11434/v1/models	Check what Ollama exposes	The exact requested model appears as an id
curl -fsS http://localhost:11434/api/show -d '{"name":"<model>:<tag>"}'	Inspect model metadata	JSON returns model information and context metadata

Checkpoint: write down the exact model ID returned by /v1/models. Use that string in your agent configuration. If the download fails, check tag spelling, free disk, and the Ollama service before moving on.

Part 1 — Connect it to Hermes Agent

Run the interactive picker outside an active session:

hermes model

Choose Custom endpoint, then enter:

• Base URL: http://localhost:11434/v1
• Model: the exact ID from /v1/models, such as llama3.1:8b
• Context length: the model’s real served value, at least 64,000 tokens

Then start a fresh one-shot chat:

hermes chat -q "Reply with exactly: OK"

Hermes command map

Command	Job	Observable success
hermes model	Select the custom endpoint and model	The picker saves the selection
hermes chat -q "Reply with exactly: OK"	Test the full route	The response is OK
hermes doctor	Diagnose provider or config trouble	The provider check identifies no blocking error

Checkpoint: Hermes returns through the local model. A slow first reply can be a cold load. A refusal to start usually means the configured context is too small; a 404 usually means the model ID does not match Ollama.

Skip the OpenClaw section and continue at Step 3.

Part 2 — Connect it to OpenClaw

OpenClaw’s canonical local-model page uses a provider block. Keep this shape and replace the model ID and context value with the ones your server reports:

models:
  mode: merge
  providers:
    local:
      baseUrl: "http://127.0.0.1:11434/v1"
      apiKey: ***
      api: "openai-completions"
      timeoutSeconds: 300
      models:
        - id: "llama3.1:8b"
          name: "Local Llama 8B"
          reasoning: false
          input: ["text"]
          cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }
          contextWindow: 65536
          maxTokens: 8192

Then select it:

openclaw models set local/llama3.1:8b

The literal *** value for apiKey is a non-secret local marker, not a real hosted-provider key.

OpenClaw command map

Command	Job	Observable success
openclaw models set local/<model>	Select the local model	OpenClaw confirms the new default
openclaw models list	Check provider discovery	local/<model> appears
openclaw infer model run --local --model local/<model> --prompt "Reply with exactly: pong" --json	Test the model without full agent context	JSON contains the requested reply
openclaw infer model run --gateway --model local/<model> --prompt "Reply with exactly: pong" --json	Test Gateway routing	JSON contains the requested reply

Checkpoint: both the direct local probe and Gateway probe work. If they pass but a full agent run fails, the model may be struggling with context or tools rather than transport. Follow OpenClaw’s local-model troubleshooting before changing unrelated config.

Step 3 — Evaluate and decide

Use three to five tasks that represent your normal work. Keep each prompt unchanged between models. Run each task three times because local output varies.

Score each response:

• 0: wrong or unusable
• 1: partly correct; needs substantial repair
• 2: correct and useful

Record median score, median latency, output length, and edge cases. A coding test should include actual code work; a tool-use test should require a structured tool call. Public benchmarks do not replace this test.

Evaluation command map

Command	Job	Observable success
time ollama run <model> "<prompt>"	Run a quick smoke test	A response and elapsed time
curl -fsS http://localhost:11434/v1/chat/completions ...	Make a reproducible API call	JSON includes choices[0].message.content
ollama ps	Confirm the model is loaded	The model appears with processor/memory details

Use the result rather than popularity:

Result	Decision
Mostly 2s, acceptable latency	Keep it for the tested job
Mostly 1s	Keep only for low-risk chat, or test a better-fitting family
Mostly 0s	Remove it from consideration
Scores vary sharply across three runs	Repeat once; if instability remains, do not make it the default

To remove a rejected download later, use ollama rm <model>. Do not delete it until results and raw responses are saved.

Have your agent do the eval

You do not have to operate this loop by hand. Install the following as a skill for any agent that can read files and use a terminal. For Hermes, create ~/.hermes/skills/evaluate-local-model/SKILL.md, paste the complete block, then run /reload-skills or start a fresh session. If HERMES_HOME is set, use $HERMES_HOME/skills/evaluate-local-model/SKILL.md instead.

---
name: evaluate-local-model
description: Evaluate or compare Ollama models on a repeatable task set
version: 1.0.0
metadata:
  tags: [local-llm, ollama, evaluation]
---

# Evaluate Local Model

## When to use
Use this skill when the user names one or more local models and a task file or task list, and wants a repeatable comparison. Requires terminal and file tools plus an Ollama-compatible server at `http://localhost:11434`.

## Inputs
Collect these before starting:
- `models`: one model name or a comma-separated list, including tags
- `tasks`: a text/Markdown file or an inline list; separate tasks with blank lines
- `goal`: coding, general chat, extraction, tool use, or another named use
- `runs`: default 3 per model/task

## Procedure
1. Read the tasks. Preserve their wording. If no tasks were supplied, ask for 3–5 representative prompts.
2. Check the server with `curl -fsS http://localhost:11434/v1/models`. Stop and report the connection error if it fails.
3. Find each requested model in the returned `id` values. For a missing model, ask permission before running `ollama pull MODEL`. Never silently download or delete a model.
4. Before evaluation, inspect each available model with:
   `curl -fsS http://localhost:11434/api/show -d '{"name":"MODEL"}'`
   Record the reported context length and family where available.
5. Create a timestamped results directory such as `local-model-eval-YYYYMMDD-HHMMSS/`. Save the exact tasks and all raw responses there.
6. For every model and task, run three sequential requests. Use Python or the shell to JSON-encode prompts; do not interpolate unescaped prompt text into JSON. POST to:
   `http://localhost:11434/v1/chat/completions`
   with `Content-Type: application/json` and this body:
   `{"model":"MODEL","messages":[{"role":"user","content":"TASK"}],"temperature":0}`
7. For each request, measure wall-clock seconds, HTTP status, and response bytes. Parse `choices[0].message.content`; if absent, record the response as an error. Save raw JSON and extracted text.
8. Grade each answer 0–2 against the task itself: 0 unusable/incorrect, 1 partial, 2 correct and useful. Do not award points for style alone. State the grading reason in one sentence.
9. For each model/task, report the median latency, median output length in characters, median score, and any edge case: timeout, malformed JSON, refusal, hallucination, truncation, inconsistent answer, or tool-call text instead of a structured call.
10. Compare models on the same tasks only. Recommend one for the stated goal, naming the quality/latency trade-off. If evidence is tied or weak, say so.

## Output
Return a Markdown report with:
- models, task source, goal, runs, endpoint, and timestamp
- one row per model/task: median latency, median output characters, median score, handled yes/partial/no, edge cases
- totals per model and a concise recommendation
- paths to the saved task copy and raw responses
- next action: keep, retry, pull a different candidate, or wire the winner into the agent

## Workflow context
If no candidate is named, first shortlist models from `https://ollama.com/library` using hardware fit, license, context length, benchmark evidence, and task fit. Download with `ollama pull MODEL` only after approval. Confirm with `/v1/models`. After choosing a winner, wire its exact model ID to the agent's OpenAI-compatible base URL; do not modify agent configuration without permission.

## Pitfalls
- Do not compare different prompts across models.
- Do not treat one run as reliable; use the median of three.
- Do not invent tokens-per-second when the API does not report token timing.
- Do not overwrite previous results or expose secrets in reports.
- Do not promote or delete a model without explicit permission.

## Verification
The run is complete only when every requested model/task pair has three attempts or a documented blocking error, raw responses are saved, and the final table can be traced to those files.

After installing it, ask your agent:

Evaluate qwen2.5-coder:7b against my coding tasks in ~/tasks/coding-bench.txt.

Or compare several models:

Compare llama3.1:8b, mistral-nemo:12b, and qwen2.5:7b on the same task set and recommend the best one for general chat.

Expect a traceable comparison containing latency, output length, task handling, edge cases, raw-result paths, and a recommendation. You can then ask the agent to apply the decision table above and pick the keeper.

Checkpoints and rollback

• Model absent from /v1/models: return to Step 1; verify the tag and download.
• Connection refused: start or restart Ollama; do not edit agent config yet.
• 404 from chat completions: copy the exact model id into the request and agent config.
• Direct probe works, agent fails: inspect the agent’s context and tool support before blaming the server.
• Evaluation is incomplete: do not compare totals. Fix the failed model/task pairs or mark them explicitly as errors.

Rollback is simple: restore the previous agent model selection. Removing a model from the agent config does not delete its Ollama files.

Variations

Variation	Change
LM Studio	Use its model ID and http://localhost:1234/v1; start the GUI server first.
vLLM	Use http://localhost:8000/v1 and the served model ID.
llama.cpp or LocalAI	Commonly use http://localhost:8080/v1; verify your launch command.
CPU-only	Begin with a smaller quantized model and include latency in the decision.
Agent skill	The skill above automates the shared evaluation loop; only provider wiring differs.

Anti-patterns

• Picking from a leaderboard alone. It measures someone else’s tasks.
• Testing each model with different wording. The comparison becomes meaningless.
• Using one run. Repeat three times and use medians.
• Configuring a guessed model name. Copy the exact ID from /v1/models.
• Silently promoting or deleting models. Save evidence and confirm the change first.

Where to go from here

Keep the task file and rerun it when a genuinely promising model appears. If this becomes routine, schedule the installed skill—but require confirmation before it downloads, deletes, or promotes a model.

Sources

• Hermes Agent — Quickstart
• Hermes Agent — Installation
• Hermes Agent — AI Providers
• Hermes Agent — Skills System
• Ollama library
• OpenClaw — Local models
• OpenClaw — Model providers
• ABS — Local LLM with Ollama: The Easy Mode