Hermes Agent Provider: OpenRouter for Cross-Vendor A/B Tests (and as a Cost Audit)

What OpenRouter is actually for in a Hermes install

OpenRouter is one routing layer that fronts many model vendors — Anthropic, OpenAI, Google, Meta, Mistral, Cohere, DeepSeek, and 40+ others — under a single API key. Two real use cases inside Hermes:

1. Cross-vendor A/B testing — when the operator wants to compare the same prompt through Claude-Sonnet, GPT-5, Gemini-2.5, Llama-4 side-by-side, OpenRouter is the fastest way to swap four providers without re-keying four APIs. Today, Hermes itself does this through hermes fallback add openrouter and per-call --provider openrouter --model <vendor>/<model>.
2. Cost audit — the live comparison dashboard at /websites/ai-model-matrix/model-matrix.html is fed by OpenRouter’s price feed. It tells the operator when a direct provider’s price drops below the OpenRouter-resold price, which is when routing through OpenRouter stops making sense.

OpenRouter is not the right place to wire your everyday primary provider. Direct providers are typically ~10% cheaper and notably faster. It also is not a free exit from the per-vendor key problem when a vendor (notably MiniMax for the M-series) charges the same rate through OpenRouter as direct — there is no upside, only the routing API call.

Step 1: sign up and fund

https://openrouter.ai/ — sign up, set billing, generate an API key. The free tier exists for low-volume testing; you will outgrow it the first time you wire it to a cron. Pricing is per-token at the published OpenRouter rate, which sits roughly 5% above each vendor’s direct rate.

Step 2: install the credential

Add OPENROUTER_API_KEY=... to /opt/data/.env. Hermes picks it up on next launch:

echo "OPENROUTER_API_KEY=sk-or-..." >> /opt/data/.env
hermes config show --provider openrouter

hermes config show lists the configured providers and confirms the key loaded. If OpenRouter does not appear in the output, double-check /opt/data/.env is sourced by the running Hermes process; some launch paths bypass the env file.

Step 3: add OpenRouter to the fallback chain

Don’t run hermes chat --provider openrouter -q "..." for everyday work — run direct vendors. Use the Hermes fallback system so OpenRouter becomes the second slot in the chain, after the primary. When the primary fails or rate-limits, Hermes steps down through the chain automatically:

hermes fallback add openrouter --priority 2
hermes fallback list

The expected output shows something like 1 anthropic_api, 2 openrouter, 3 ollama_local. If your Anthropic key is rotated or the upstream returns a 529, the runtime tries OpenRouter before it tries local Ollama. This is the order you’d wire if a single vendor outage must not stop the operator’s chat.

Step 4: A/B test through it

When you actually want the cross-vendor compare, do it explicitly per-call, in the same chat session:

hermes chat --provider openrouter --model anthropic/claude-sonnet-4.5  -q "Summarize today's inbox"
hermes chat --provider openrouter --model openai/gpt-5               -q "Summarize today's inbox"
hermes chat --provider openrouter --model google/gemini-2.5-pro       -q "Summarize today's inbox"

OpenRouter model strings follow <vendor>/<model> — confirm against the live list at https://openrouter.ai/models before each run; new models land there daily. Compare the three outputs for the same prompt, log the result, and pick a vendor per use-case. This explicit-compare workflow is the one that genuinely justifies having the OpenRouter key on file.

The cost audit

OpenRouter’s per-vendor feed is a free reference for what direct pricing should be. When the live dashboard shows a vendor’s OpenRouter-resold price going above the direct price, that’s a signal to flip the direct path on for that vendor — running that vendor through OpenRouter stops making sense. The dashboard at https://agenticbotsitter.com/websites/ai-model-matrix/model-matrix.html is rebuilt every morning by a cron; treat any “OpenRouter ★” tag as the recommended-path flag. The price lag from OpenRouter’s feed is typically a few hours; that’s good enough for human decisions, bad enough that you cannot use it for hot-path price optimization.

What not to do

• Don’t put OpenRouter first in the fallback chain if direct providers work for the relevant vendors — that’s a 5% markup you didn’t need.
• Don’t use the free tier for any production cron — it rate-limits within an hour.
• Don’t store the API key in plaintext markdown — use /opt/data/.env only.
• Don’t run --provider openrouter --model <vendor>/<model> thinking it saves money on the vendor’s direct traffic. For most vendors the price is identical or higher through OpenRouter. The cost audit table shows you this.
• Don’t switch models inside a cron prompt without testing on --dry-run first.

Verification checklist

#	Command	What it proves
1	hermes config show --provider openrouter	OpenRouter configured
2	hermes fallback list	OpenRouter is in the fallback chain, second slot
3	hermes chat --provider openrouter --model anthropic/claude-sonnet-4.5 -q "Reply with OK"	Cross-vendor access works
4	curl https://agenticbotsitter.com/websites/ai-model-matrix/model-matrix.html	Cost audit shows current pricing
5	~/.hermes/state/sessions.db	Last chat’s provider recorded for audit

Sources

• OpenRouter docs
• Hermes Agent providers
• ABS live dashboard: AI Model Matrix
• ABS companion: Hermes Agent: Switching Providers Mid-Run

Last verified: 2026-07-14 against Hermes v0.18.2 (2026.7.7.2) and OpenRouter pricing snapshot 2026-07-14. OpenRouter model strings change frequently; verify against https://openrouter.ai/models before any production wiring.