Provider Rate Limits and the Fallback Rule: When Hermes Must Switch vs When It Must Hold

Every provider bills in tokens, but every provider bills tokens inside a different cage. Anthropic gates you on requests-per-minute plus tokens-per-minute; OpenAI adds a daily cap; OpenRouter inherits whichever limits the upstream vendor imposes and layers a 402 budget cap on top; local Ollama has no per-token cage at all — only VRAM and a wall clock. A fallback chain that does not understand the difference falls back at the wrong time, in the wrong order, and sometimes twice for the same failed turn. This guide covers the three rate-limit shapes, the chain Hermes is configured with, when the chain must fire, when it must hold, and the rule that keeps a failed turn from being billed twice.

The three rate-limit shapes

Provider APIs report throttling in three distinct currencies. A working fallback chain treats them differently.

Shape	What it caps	Window	Typical workload	Header to read
RPM — requests per minute	Distinct POST /v1/chat/completions calls	Sliding 60s	Burst workloads, parallel cron fan-out	x-ratelimit-limit-requests, Anthropic’s anthropic-ratelimit-requests-limit
TPM — tokens per minute	Input + output tokens across all calls	Sliding 60s	Long-context summaries, doc-embedding sweeps	x-ratelimit-limit-tokens, Anthropic’s anthropic-ratelimit-tokens-limit
RPD — requests per day	Distinct API calls against a tenant key	Calendar or rolling 24h	Hourly-batch crons, daily newsletters	OpenAI resets at 00:00 UTC; Mistral exposes monthly-requests

RPM and TPM reset on a sliding window; RPD usually resets on a calendar boundary. The right reaction is different for each. An RPM cap lifts in 60s; a TPM cap also lifts in 60s but only if you stop sending more tokens; an RPD cap does not lift until midnight UTC. Retrying an RPD-capped request through the same provider is guaranteed to fail — walking the fallback chain is the only correct move.

Hermes’ retry layer (api_max_retries, default 3) handles short-window blips on its own, with Retry-After honored and exponential backoff plus jitter for the rest. The fallback chain is for what the retry layer cannot fix: a primary still failing after the budget is spent, an RPD cap that will not lift today, a vendor-side 5xx storm, or a key that is no longer valid.

The fallback chain config

The ABS / Hermes convention is a three-tier chain declared once in ~/.hermes/config.yaml:

# ~/.hermes/config.yaml
model:
  provider: anthropic
  default_model: claude-sonnet-5-latest

fallback_providers:
  - provider: openrouter
    model: anthropic/claude-sonnet-4.5
  - provider: local
    base_url: http://127.0.0.1:11434/v1
    model: qwen2.5-coder:32b

The shape is anthropic → openrouter → local. Slot 1 is the direct, fastest, cheapest path for the dominant job. Slot 2 is a cross-vendor router that buys the same family through a different billing relationship when the direct path is the problem. Slot 3 is local inference — no per-token cost, no per-minute cage, only VRAM and patience.

Three rules when writing the chain:

1. Order is failure order, not preference order. The chain is walked only when the previous slot fails. Putting local Ollama at slot 1 because it is free buries every request behind 40-90s per-turn latency on anything larger than a 7B model.
2. Match model tier across slots if the job depends on capability. If your primary is Claude Sonnet for structured output, your slot 2 should also be Claude Sonnet through OpenRouter, not gpt-4o-mini. A silent quality drop is worse than a logged failover.
3. Pin the local slot to a model you have actually pulled. qwen2.5-coder:32b is fine; claude-opus-4-8-latest returns a 404 model not found that the chain treats as a connection error and then has nothing left to try.

The chain is re-read on each chat invocation. Editing the config file while the gateway is running takes effect on the next turn; no Hermes restart is needed.

When the chain MUST fire

Walk the chain when the primary returns one of these:

• HTTP 429 with no Retry-After, or a Retry-After larger than 30s. Rate-limited and the retry layer’s jitter will not save you.
• HTTP 529 (Anthropic OverloadedError). Provider-side outage; retries within the same minute only make it worse.
• HTTP 402 from OpenRouter. Budget cap; the auxiliary client marks it unhealthy for 60s and subsequent calls skip it organically.
• HTTP 5xx for 3 consecutive attempts. Transient cluster failure that survived the retry budget.
• Connection timeout or DNS failure to the primary. Verify with curl -I https://api.anthropic.com/v1/messages before declaring it local.
• Authentication failure mid-turn. Key rotated, OAuth refresh failed, or the slot you are falling back to has no key on file.

The chain does not think; it walks.

When the chain MUST NOT fire

Five failure shapes where walking the chain wastes tokens or time:

Shape	Why hold	What to do instead
HTTP 400 / bad_request	The provider understood you and said no. The next provider will, too.	Fix the prompt. A 400 from Anthropic is a 400 from OpenRouter-routed Anthropic and any local server.
HTTP 401 first time on a new key	The key is bad, expired, or not yet active. The next slot will accept — and bill.	Fix the credential, then retry the primary.
content_policy / safety block	Judgment on the prompt, not a network problem.	Edit the prompt; run again on the primary.
Self-inflicted retry loop	A cron that retries forever, swaps on every loop, burns 100x budget recovering from one error.	Cap retries. Treat 3 failed chains in 10 minutes as a single failure.
Operator forced --provider X	You asked for X. Walking the chain behind your back contradicts the request.	Honor the override; document it in session notes.

The chain exists to recover from network and capacity problems, not to paper over prompt or credential bugs. The fastest wrong fallback is the one that does not happen.

The fallback must never pay twice rule

One sentence:

If slot N failed for a reason the retry layer would have fixed given enough budget, do not bill slot N+1 for the same work.

Three corollaries make the rule operationally true:

1. The retry layer must finish before the chain walks. api_max_retries (3 by default) plus jitter must exhaust on the primary before slot 2 is considered. If the primary was a 5s blip, the chain must not see it.
2. Idempotency travels with the turn, not the provider. If the failed turn on slot 1 already created a side effect (a logged message, a cached tool result, a partially-written file), slot N+1 resumes from that state. The assistant turn that shipped a tool call does not run the tool call again on the new provider.
3. Each chain slot carries its own budget. A three-slot chain with one shared cap is two slots paying twice; the chain becomes a budget multiplier on outage days. Set per-slot caps in provider.<name>.monthly_cap_usd; let each slot fail independently.

Violating the rule looks like this in the logs: 12:00:01 anthropic 429, 12:00:02 openrouter 200 (12,400 input, 380 output), 12:00:03 anthropic retry 200 (12,400 input, 380 output after backoff). Two providers billed for one turn. The fix is to gate the chain on retries_exhausted, not on first_failure. Hermes implements this correctly; an operator who overrides the chain with a manual --provider hop can break it.

Verification checklist

#	Command	What it proves
1	hermes fallback list	Chain shows anthropic → openrouter → local in order
2	hermes chat -q "Reply with OK"	Primary path works end-to-end in <5s
3	Run hermes chat --provider anthropic -q "Reply with OK", --provider openrouter, --provider local in sequence	Every slot individually reachable
4	tail -50 ~/.hermes/logs/agent.log | grep -E "fallback|unhealthy|429|overloaded"	Last chat logged the right provider and the right failure code if any
5	hermes config show | grep monthly_cap_usd	Each slot has a per-provider soft cap configured

Sources

• Hermes Agent fallback providers
• Hermes Agent provider docs
• Hermes CLI commands reference
• Anthropic API rate limits
• OpenAI API rate limits
• ABS companion: Hermes Agent Provider: Anthropic API
• ABS companion: Hermes Agent Provider: OpenRouter
• ABS companion: Hermes Agent Provider: Local OpenAI-Compatible
• ABS companion: Hermes Agent: Switching Providers Mid-Run When a Provider Fails

Last verified: 2026-07-15 against Hermes v0.18.2 (2026.7.7.2). The 60s unhealthy cooldown and 3-retry default are from a clean install; older installs may carry stale overrides.