Temperature, Top-P, and the Other Sampling Knobs: When to Touch Them and When to Leave Them Alone

Every chat-completions request has a small panel of knobs you can turn: temperature, top_p, top_k, frequency_penalty, presence_penalty, max_tokens, and seed. Every team that wires up an LLM eventually argues about them. The argument is almost always wrong, because most calls do not need any of these knobs turned at all. The defaults the provider ships with were chosen to make the model behave well across the broadest range of prompts. Touching them without a reason is the most common way to ship a worse model than the one you paid for. This guide covers what each knob actually does, when one of them legitimately matters, and the three anti-patterns that show up in production code more often than they should.

The seven knobs, in plain language

Knob	Range	Default (OpenAI/Anthropic)	What it really does
temperature	0.0–2.0	1.0 (OpenAI), 1.0 (Anthropic)	Rescales the logit distribution before sampling. 0 makes the model greedy (argmax). Higher values flatten the distribution and surface less-likely tokens.
top_p	0.0–1.0	1.0	Nucleus sampling: keep the smallest set of tokens whose probabilities sum to top_p. 1.0 = no truncation.
top_k	1–∞	not exposed (OpenAI/Anthropic); some open models default to 40	Hard cap on the candidate set: keep only the k highest-probability tokens at each step.
frequency_penalty	-2.0–2.0	0	Subtracts a constant from a token’s logit every time it has already appeared. Positive values discourage repeats.
presence_penalty	-2.0–2.0	0	Subtracts a constant the first time a token has appeared. Positive values push the model toward new vocabulary.
max_tokens	1–model max	unset	Caps the response length. The model emits an end-of-turn token when it thinks it is done; this is a ceiling, not a target.
seed	integer	unset	A deterministic seed for sampling. Same prompt + same seed + same model = same output (modulo provider implementation).

Two of these are universal across providers (temperature, max_tokens). Two are OpenAI-shaped and pass through OpenRouter to most upstreams (frequency_penalty, presence_penalty). One is more common on open-weight models exposed via Ollama and friends (top_k). One is the quietly powerful one (seed). Anthropic does not accept top_k, frequency_penalty, or presence_penalty at all — its provider rejects them with a 400.

The default-everything rule

One sentence:

If you cannot name the concrete failure that a non-default value will fix, leave the knob alone.

Three corollaries:

1. Defaults are tuned on the same prompts you are sending. The vendor’s eval suite is broader than yours. Their default is the value that scored best across the widest mix of code, chat, and summarization. Your one weird workload is not the eval suite.
2. Knobs interact. Raising temperature to 0.7 and then capping top_p at 0.9 is two changes at once — when the result is worse, you do not know which one did it. Change one knob, re-run the eval, then change the next.
3. Defaults compose well with provider-side retry. The retry layer, the fallback chain, and the response caching all assume default sampling. Non-default values can quietly defeat caching (different seed every turn, different temperature per user) without the operator noticing.

The default-everything rule is not “never touch a knob.” It is “the burden of proof is on the knob, not the default.”

When each knob actually matters

Most days, none of them matter. When one does, it is because the task has a specific shape that the default sampling does not handle well.

Task shape	Knob that helps	Value to try	Why the default fails
Deterministic re-runs (CI evals, golden-file tests, regression suites)	seed	a fixed integer, logged with the request	Defaults introduce run-to-run variance, so golden files drift and CI flakes.
Strict formatting (JSON-only, regex-validated, schema-constrained)	temperature	0	Defaults produce } followed by "Sure, here is..." more often than people expect.
Long-form creative writing where defaults feel flat	temperature	0.7–0.9	Defaults are tuned for average helpfulness; creative work sometimes needs more spread.
Avoiding token loops in long completions	frequency_penalty	0.1–0.4	A handful of tokens can dominate the probability mass; a small penalty breaks the loop without changing style.
Generating lists of distinct items (brainstorms, alternative phrasings)	presence_penalty	0.1–0.6	Defaults favor the high-probability token, which is the one the model already said. Presence penalty forces the model to find a new word.
Capping a runaway response (cron summaries, JSON blobs)	max_tokens	the smallest value that fits the schema	Without a cap, a chatty model produces a 4000-token answer when 200 would do.
Open-weight model via Ollama where top_k defaults to 40	top_p set to 1.0 and top_k left alone, or vice versa	pick one	Running both creates a redundant cap; whichever is tighter wins, the other is dead weight.

The pattern: each knob has one job, and the job only shows up when the task violates a specific assumption the default makes.

The three anti-patterns

Three mistakes show up often enough to be worth naming.

Anti-pattern 1: temperature: 0.7 for code generation

This is the most common one. A team wires up a code-completion endpoint, somebody on the team reads a blog post that says 0.7 is “creative,” and the YAML gets it. The result: tests that pass on Monday fail on Tuesday because the same prompt produces different function bodies. Code generation is a deterministic task — the right function is one of maybe three candidates, and the wrong one is the long tail. For code, JSON, regex, SQL, and any other grammar-shaped output, temperature: 0 (or close to it) is the right answer. Save 0.7 for marketing copy.

Anti-pattern 2: setting top_p: 0.95 and top_k: 40 on the same request

top_p and top_k both restrict the candidate set. Set both and the model applies whichever is stricter. Two restrictions, one job, half the time you are not sure which one is doing the work. Worse: if the request worked at top_p: 0.95 alone, adding top_k: 40 may now cap your candidate set at 40 even when the next 41st token would have been the right one. Pick one. The OpenAI / Anthropic docs recommend changing one at a time; they recommend not changing both.

Anti-pattern 3: setting seed to chase variance

seed is for removing variance, not producing it. A team that is unhappy with the diversity of outputs should reach for temperature, not seed. seed: 42 does not guarantee the model will pick the canonical answer; it guarantees the same prompt produces the same answer on every run. To get a spread of candidates, run the prompt N times with temperature: 0.7 and seed unset, or use a small frequency_penalty. Mixing the two responsibilities (deterministic with seed, then varying temperature per run) defeats both goals: you no longer have determinism and you do not have clean variance control either.

Verification checklist

#	Command	What it proves
1	`grep -nE “temperature	top_p
2	hermes chat -q "Return exactly {\"ok\": true}" --temperature 0	temperature: 0 produces valid JSON with no trailing prose
3	hermes chat -q "Write a haiku about Octocat" --seed 7 --temperature 0	Same seed returns the same haiku across two runs
4	hermes chat -q "List 5 distinct one-word titles for a blog post" --presence-penalty 0.4	The five titles share no words
5	hermes chat -q "Summarize this in 30 words" --max-tokens 60	Response stays under the cap even when the model “wants” to write more

Sources

• OpenAI Chat Completions API reference — sampling params
• Anthropic Messages API — temperature and sampling
• OpenRouter request parameters reference
• Hermes Agent provider configuration reference
• ABS companion: Provider Rate Limits and the Fallback Rule
• ABS companion: Hermes Agent Provider: Anthropic API
• ABS companion: Hermes Agent Provider: OpenRouter
• ABS companion: Hermes Agent: Switching Providers Mid-Run When a Provider Fails

Last verified: 2026-07-15 against Hermes v0.18.2 (2026.7.7.2). Defaults quoted are OpenAI’s gpt-4o/4.1 and Anthropic’s claude-sonnet-4.5 defaults; OpenRouter inherits the upstream defaults. Provider-specific behavior can shift between model versions — re-verify with hermes config show before relying on a default in a regression suite.