Personal Model Benchmarks: Stop Chasing a Winner and Assign Each Model a Job

The Problem With Chasing a Winner

You have work to do: a research brief, a client writeup, code, a strategic decision. You also have model announcements, leaderboards, and breathless takes telling you the new release is finally the one. Act on every claim and you spend the week re-running the same prompt across interfaces. Ignore them and you quietly ride an old default.

A small standing benchmark that fits how you actually work is the way out. Not to crown a universal best model, which a personal suite of ten to twenty cases and three trials cannot do. The point is narrower: pick a default per task family, change it when your own evidence moves, and stop reading leaderboards as if they were about you. Anthropic’s demystifying evals for AI agents makes the basic move explicit: outputs vary run to run, so any personal signal needs multiple trials and a recorded transcript. NIST’s AI Risk Management Framework frames evaluation around context, intended use, uncertainty, and ongoing testing. The Chatbot Arena paper shows how anonymous pairwise voting can reduce identity-tainted scoring.

The phrase to keep taped to the monitor: a benchmark that cannot change a real default is only a hobby.

What You Are Measuring

Not raw intelligence. You are measuring fit between a configuration and a task family, on a specific date, on your inputs. The configurations you compare can be different models, the same model at different settings, or different harnesses wrapped around the same model. The task families that pay rent for a solo operator are research, writing, strategic thinking, and building, the four the AIDB transcript on standing personal benchmarks flags as worth tracking.

Decide before you test which family each case belongs to, and which family is allowed to override another when scores conflict. Opinionated work like strategy and personal voice should not be knocked off its default by a coding benchmark, even if your coding suite loves the challenger. The evaluation best-practices guide pushes the same idea at a more technical level: task-specific evals, logs of everything, automated grading where it validly captures success, and human judgment where it cannot.

A small but important caveat on outputs. Two configurations that look identical in their visible controls may not be running on identical compute, and you will not see that from the outside. Treat your results as evidence about the specific configuration you tested on the date you tested it. Re-run when either side changes.

Steps

Step 1: Choose the decision the benchmark must move

Write one sentence: “If the numbers come out X, I switch the default for Y.” If you cannot write that sentence, the benchmark is decorative. The sentence can be small, like “the new challenger takes over research summaries only if it loses zero of ten cases to my current model.” Be honest that you are allowed to end up with a single ecosystem across all four families. That is a rational outcome, not a failure, if it is the answer your evidence supports.

Step 2: Select 10 to 20 cases from real recurring work

Pull cases from your own backlog, not synthetic prompts. Start with a suite of 10 to 20 cases that you can run and inspect. For each case, write down what success looks like before you run anything. Mix typical cases, edge cases, and one or two adversarial cases you would rather not have to handle again. The evaluation best-practices guide names those categories; they help expose failures flat synthetic sets can hide.

Sanitize inputs before they leave your machine. Strip names, addresses, client identifiers, contract numbers, and any other private material, or run the case against a redacted analogue. Your benchmark is not a reason to widen your data boundary.

Step 3: Define gates and rubrics

For deterministic work, a binary gate is fine. Does the code pass the existing test suite? Does the JSON parse? Does the SQL run? For subjective work, write a short rubric with three to five named dimensions, such as factual accuracy, structure, voice, and one thing specific to the task. Decide in advance what score wins a tie, and allow a “both bad” verdict so you can demote both rather than be carried by a coin flip. For comparative runs, do not look at which model produced which output while you are scoring. Pairwise blind comparison, with model identity revealed only after the vote, is the lesson the Arena paper actually teaches.

Step 4: Lock the run protocol

Three trials per case is a practical starting point, not a statistical guarantee. Pick a number and write it down, then hold it constant across configurations so comparisons stay comparable. Fix the prompt text. Fix the system prompt. Fix the temperature if you can. Record generation time, response time, retries, and any visible cost or plan impact. End-to-end task burden matters more than a single latency number, because your real cost is generation plus review plus edit plus any retry.

Step 5: Run and compare

Run your current default first against the full set. Then run the challenger. Score blind, then compare. If you cannot blind yourself because one configuration has a recognisable voice, accept the noise and say so out loud in your notes. Treat any LLM judge as an assistant reviewer, not an objective grader; it has the same style blind spots you do.

Step 6: Make model-task cards

Write one card per configuration per task family. Each card lists the configuration as tested, the date, the cases and trial count, the pass rate on deterministic gates, the rubric summary on subjective ones, the median end-to-end time, and a one-line verdict. The card is what you actually look at when the next announcement drops. Anything not on the card is not part of your evidence.

Step 7: Refresh on change, not on hype

Re-run when the configuration you depend on changes, when a task family starts producing more edits than it used to, or when a new contender arrives with a specific claim you can translate into a case. Do not re-run on vibes. NIST’s framework frames evaluation as ongoing; treat that as a reason to retest on change, not as a weekly chore.

Failure Modes

Treating the personal suite as a universal ranking. Ten cases and three trials cannot tell you which model is best in general. It can tell you which configuration to default to, on your work, on this date.

Reading the same-prompt test as if it were real-work evidence. A canonical prompt that both models see identically is useful, but it is not the same as a full workflow that includes review and edit. Keep both kinds of test, label them clearly, and do not let the cleaner number win the argument the dirtier number is about.

Trusting an LLM judge for voice. Voice is personal. The judge’s preferences are not yours. Use pairwise blind human scoring for anything that lives in your byline, and treat automated judging as a coarse filter only.

Assuming visible settings reveal compute. They do not. Your evidence is about the configuration you tested, full stop.

Letting private material into the benchmark. Sanitize first.

Refusing to settle. Stop re-running to chase a margin your evidence cannot measure.

Done means

You can point at a written one-sentence decision rule that the suite can actually move, plus a stack of dated model-task cards, one per configuration per family. Each card lists the cases, the trial count, the gates or rubric summary, the end-to-end time, and a verdict you wrote while blind. The next time a new release arrives, you spend an afternoon, not a week. You add or replace cases from real work, you do not redesign the suite. When the configuration you depend on changes, you know because your cards are dated. If you cannot write the decision rule, the suite is a hobby, and that is the phrase to remember.

What this article does not cover

Enterprise procurement, vendor selection, or contracts. Public leaderboard construction, statistical sampling theory, or how to design a benchmark that survives academic peer review. Safety and policy evaluation at the organizational level, which lives under frameworks like NIST AI RMF rather than personal task fitness. Pricing comparisons, rate limits, context windows, or any current model rankings. A claim that this approach will surface a universal best model, because ten cases and three trials will not. Any claim of personally tested results; the steps above are advice for you to run, not results from me running them.

Related guides

• AI Model Matrix Quickstart
• When Labs Control the Model and Harness
• How to Read an LLM Pricing Table Without Fooling Yourself

Sources

• Anthropic, Demystifying Evals for AI Agents
• OpenAI, Evaluation Best Practices
• NIST AI Risk Management Framework (AI 100-1)
• Chatbot Arena: An Open Platform for LLM Evaluation
• AIDB, Standing Personal Benchmarks Transcript