Hermes Agent Provider: Local OpenAI-Compatible (vLLM, Ollama, LM Studio)

Why a local OpenAI-compatible provider

Paid providers — Anthropic, OpenAI, Google, OpenRouter — bill per token. Local inference flips that: hardware already paid for, weights already downloaded, every chat at $0.00/Mtok. The catch is hardware. A 70B model on a Mac mini is slow; the same model on a VPS with an H100 is not.

Hermes treats any OpenAI-compatible HTTP endpoint as a provider. Ollama, vLLM, LM Studio, llama.cpp’s server mode, text-generation-inference, LocalAI all expose the same /v1/chat/completions shape. Hermes points at one with --base-url, names a model with --model, and the chat works like a paid provider — on the metal under your desk.

The setup shape:

hermes chat --setup local \
  --base-url http://localhost:11434/v1 \
  --model qwen2.5-coder:32b \
  -q "Reply with OK"

Server up and model pulled: that call round-trips in 5s on a GPU box, 30-90s on a CPU Mac mini. Everything below is about picking the right server for the right box.

The base-url pattern

The OpenAI-compatible contract has three pieces Hermes relies on:

1. GET /v1/models — list of pulled models; used for --model autocomplete and validation.
2. POST /v1/chat/completions — the chat call; Hermes sends the OpenAI JSON shape, server returns the OpenAI JSON shape.
3. Bearer auth via Authorization: Bearer <key>; local servers accept any non-empty string.

Server	Default base URL
Ollama	http://localhost:11434/v1
LM Studio	http://localhost:1234/v1
vLLM	http://localhost:8000/v1

The same flags work over LAN or WAN:

hermes chat --setup local --base-url http://127.0.0.1:8000/v1   --model meta-llama/Llama-3.1-70B-Instruct  # same machine, different port
hermes chat --setup local --base-url http://192.168.1.50:8000/v1 --model Qwen/Qwen2.5-Coder-32B-Instruct    # LAN GPU box
hermes chat --setup local --base-url https://llm.example.com/v1  --model qwen2.5-coder:32b                 # via Cloudflare Tunnel

--base-url is the only thing that makes a local backend a provider on Hermes. --setup local persists the base URL and model into ~/.hermes/config.yaml.

The token-budget trade-off

Local inference is free only at the per-token layer:

Box	Model	Tokens/sec	Cost	Verdict
Mac mini M4 (16 GB)	8B Q4_K_M	8-15	$0	Fine for drafts, slow for cron
Mac mini M4 Pro (48 GB)	32B Q4_K_M	5-10	$0	Viable; watch thermals
VPS, no GPU ($7/mo KVM)	8B Q4_K_M	4-8	$0	Marginal
VPS, single L4 (RunPod)	32B full	40-70	$0 (~$0.40/hr)	Sweet spot for cron + chat
VPS, single H100 80GB	70B full	80-150	$0 (~$2/hr)	Frontier-class local

Break-even: more than ~2 hours/day of inference, a $0.40/hr GPU spot is cheaper than a $4,000 Mac Studio. Less than 30 minutes/day, the Mac mini already on your desk is the right answer.

Latency, not cost, is the real reason local loses to paid providers. Anthropic Sonnet at ~80 tok/sec on the wire beats a CPU-bound Mac mini at 5-15 tok/sec even when free. For real-time chat this matters; for batch cron, it doesn’t.

Case study 1: LM Studio on macOS

The easiest local provider — desktop app, no terminal. Download from https://lmstudio.ai/ (universal binary, Apple Silicon native, no Rosetta). Search, pick a model (e.g., Qwen2.5-Coder-32B-Instruct-GGUF), Download, then open the Local Server tab, pick the model, port 1234, Start Server. Verify with curl http://localhost:1234/v1/models and wire Hermes:

hermes chat --setup local --base-url http://localhost:1234/v1 \
  --model qwen2.5-coder-32b-instruct-gguf -q "Reply with OK"

LM Studio’s Metal backend uses Apple’s GPU and Neural Engine. On a Mac mini M4 Pro with 48 GB, a 32B Q4_K_M model loads in ~20 seconds and runs at ~8 tokens/sec — fine for interactive chat, too slow for cron.

Downsides: the app must stay open (no headless daemon mode in the current build); sleep kills the server (disable Mac sleep while cron is running); no automatic model unload (use Eject in the Local Server tab to free RAM).

Case study 2: Ollama on a VPS

Ollama on a Linux VPS is the production-grade local provider: systemd-managed, automatic crash restart, on-disk model cache, OpenAI shim at :11434. Install: curl -fsSL https://ollama.com/install.sh | sh && ollama pull qwen2.5-coder:32b. Ollama’s OpenAI shim is built-in; no extra flag.

Wire Hermes from your laptop: hermes chat --setup local --base-url http://vps.lan:11434/v1 --model qwen2.5-coder:32b (LAN), or via the Cloudflare Tunnel exposed URL.

Ollama runs on CPU out of the box. On a $7/mo KVM VPS, expect 4-8 tokens/sec on an 8B model — too slow for per-minute cron, fine for hourly. With a GPU (RunPod, Vast.ai, Lambda Cloud), Ollama picks it up via CUDA and the same 8B model hits 80+ tokens/sec.

Gotcha: Ollama’s default cache lives on the small root partition on most VPSes. Before the first pull, symlink /usr/share/ollama/.ollama/models to a data volume — otherwise the disk fills on the first ollama pull of a 70B model.

Case study 3: vLLM on a remote GPU box

The throughput-optimized local provider. Same OpenAI-compatible surface, designed for batched high-throughput serving on dedicated H100/A100 GPU boxes.

vllm serve Qwen/Qwen2.5-Coder-32B-Instruct --host 0.0.0.0 --port 8000 --max-model-len 8192 --gpu-memory-utilization 0.92

The first vllm serve downloads the model, compiles CUDA kernels, warms the KV cache — expect 2-5 minutes before the first chat returns. After that, throughput is the highest you’ll see from any local server: 80-150 tokens/sec on a single H100 for a 32B model.

Wire Hermes: hermes chat --setup local --base-url http://gpu-box.lan:8000/v1 --model Qwen/Qwen2.5-Coder-32B-Instruct -q "Reply with OK".

vLLM is the right pick when the box exists solely to serve LLM traffic. Downsides: heavier install (CUDA toolkit, ~10 GB of pip deps), slower boot than Ollama, fixed VRAM cost — a 70B model at full precision needs ~140 GB. For 32B-class models, a single H100 80GB is the sweet spot. The OpenAI surface at :8000/v1 is complete (streaming, function calling, embeddings, /v1/models); the model name is the HuggingFace repo path.

Mixing local with paid providers

A real Hermes install rarely runs on local-only. The fallback chain pattern:

# ~/.hermes/config.yaml
fallback_providers:
  - provider: openrouter
    model: anthropic/claude-sonnet-4.5
  - provider: openai
    model: gpt-5-mini
  - provider: local
    base_url: http://gpu-box.lan:8000/v1
    model: Qwen/Qwen2.5-Coder-32B-Instruct

Three roles a local provider plays in a multi-provider setup:

1. Cheapest fallback slot. When paid providers are rate-limited or down, the local box catches the load. Free, but slow — only kicks in for cron or non-interactive jobs.
2. Privacy-tier provider. Tasks that must not leave the operator’s hardware (medical drafts, financial analysis, internal R&D) pin to local and skip cloud entirely.
3. Always-on backup. Even with paid providers first, the local box is the one that survives an outage of every paid API simultaneously.

The order matters. Putting local first means every chat waits for local inference; putting it last means local is the catch-all when nothing else works. The right order for a single-operator install is paid-first, local-last.

What not to do

• Don’t point --setup local at an endpoint you don’t control — --base-url is bearer-authenticated, and a paid-provider URL on the local wire will silently bill you.
• Don’t pull a 70B model on a 16 GB Mac; the daemon OOMs mid-prompt. Stay at 8B on 16 GB; 32B at Q4_K_M needs 24+ GB.
• Don’t run LM Studio’s app-embedded server as the only path. The app must stay open; the screen must not sleep. Use Ollama or vLLM for unattended jobs.
• Don’t typo the model name. Hermes validates against /v1/models before sending the chat; a typo returns model_not_found.
• Don’t assume a CPU VPS is fast enough. $7/mo KVM at 4-8 tok/sec on an 8B model is fine for hourly cron, too slow for chat.
• Don’t confuse --setup local with --provider local. Same config keys; --setup is clearer in cron logs.

Verification checklist

#	Command	What it proves
1	curl http://localhost:11434/v1/models (or :1234 / :8000)	Local server reachable, model loaded
2	hermes config show --provider local	base-url and model persisted
3	hermes chat --provider local -q "Reply with OK"	End-to-end chat works on local
4	nvidia-smi (GPU box only)	VRAM is allocated
5	ollama list / lms ls / vllm serve log	Daemon alive

Sources

• Hermes Agent provider docs
• Ollama — OpenAI compatibility
• LM Studio — Local Server
• vLLM — OpenAI compatible server
• ABS companion: Hermes Agent: Switching Providers Mid-Run
• ABS companion: Hermes Agent on a Mac mini M4

Last verified: 2026-07-14 against Hermes v0.18.2 (2026.7.7.2). Local provider surface validated against curl /v1/models on three backends.