Local LLM with Ollama: The Easy Mode (L7)

Why Ollama is the easy mode

Three credible ways to run a local LLM in 2026: LM Studio (desktop app), vLLM (high-throughput serving on a GPU box), and Ollama — one daemon, one CLI, an OpenAI-compatible HTTP shim. Ollama is the easy mode because the whole install is three commands and the daemon is one process you can forget about.

The contract: pull a model, the daemon caches it. Hit http://localhost:11434, it answers in the OpenAI /v1/chat/completions shape, so every OpenAI-compatible tool talks Ollama for free. No Python venv. No CUDA toolkit. No systemd unit to write — Ollama ships one.

The three install steps

On Linux, macOS, or Windows (via WSL2), the whole path is curl / pull / serve:

# 1. Install the daemon (Linux). macOS: download from ollama.com. Windows: install WSL2 then run this.
curl -fsSL https://ollama.com/install.sh | sh

# 2. Pull a model — files land in /usr/share/ollama/.ollama/models by default
ollama pull llama3.1:8b

# 3. Serve. The install script already starts ollama serve; this is the manual form.
ollama serve
curl http://localhost:11434/v1/models   # verify

That is the whole thing. The first /v1/chat/completions call loads the model into RAM or VRAM; subsequent calls hit the warm cache at full speed. On a GPU box the first token lands in ~1s; on a CPU-only Mac mini, first token is 5-15s and steady-state 8-15 tokens/sec for 8B. Verify the OpenAI shim:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.1:8b","messages":[{"role":"user","content":"Reply with OK"}]}'

A 200 response with "content":"OK" proves the daemon is up, the model is loaded, and the OpenAI wire works.

The model-shorthand table

The Ollama library uses family:size shorthand. Picking the right size is 90% of “is this going to work on my box”:

Shorthand	Params	RAM (Q4_K_M)	Where it fits
8B	~8B	~6 GB	16 GB laptops, $7/mo VPS, edge
14B	~14B	~10 GB	16 GB Mac, mid VPS, Raspberry Pi 5 with swap (slow)
32B	~32B	~22 GB	32 GB Mac Studio, single L4 GPU, RunPod spot
70B	~70B	~48 GB	64 GB Mac Studio, single A100 80GB, H100

A Q4_K_M model wants roughly 0.75x its parameter count in GB of unified memory. A 70B Q4_K_M at ~48 GB does not fit on a 32 GB Mac — use the 64 GB Mac Studio, an A100 80GB, or drop to Q3_K_M at ~36 GB and accept the quality hit.

Picks by tier: 8B: llama3.1:8b, qwen2.5:7b, mistral:7b, phi3:mini. 14B: qwen2.5:14b, llama3.1:13b. 32B: qwen2.5-coder:32b, qwen2.5:32b, deepseek-coder-v2:16b. 70B: llama3.1:70b, qwen2.5:72b, deepseek-r1:70b, mixtral:8x22b (MoE, ~140B effective at 70B VRAM).

The :tag conventions

After the family and size, the second half of the model name is the tag — and tags are where Ollama hides the quantization level, the variant, and the context length:

llama3.1:8b-instruct-q4_K_M
└──┬───┘ └┬┘ └──┬────┘ └┬─────┘
 family   size variant   quant

The three tag parts that matter:

Part	Common values	What it means
Variant	instruct, chat, coder, base	base continues text — not useful for chat. instruct / chat are post-trained for instruction-following. coder is post-trained for code. Always pick instruct (or a specialist) for chat.
Quant	q4_K_M, q4_0, q5_K_M, q6_K, q8_0, fp16	Lower bits = smaller download, less RAM, faster, but dumber. q4_K_M is the default and the right pick 90% of the time. q8_0 is the highest-quality quant that still halves the file size. fp16 is full precision — only sane on a frontier GPU.
Context	sometimes :32k, sometimes omitted	Larger context costs more VRAM and slows each token. Default is 2k-8k; bump with num_ctx in the modelfile if your box can hold it.

K-quants (q4_K_M, q5_K_M, q6_K) are llama.cpp’s k-quantized medium variants — slightly smarter than q4_0 at the same size. The trailing letter is the variant tier: _S (small), _M (medium), _L (large). q4_K_M is the median.

Defaults that hide in ollama pull: no tag (ollama pull llama3.1) resolves to :latest, which is q4_0 for legacy models and q4_K_M for newer ones — explicit is safer. Adding -instruct explicitly (llama3.1:8b-instruct-q5_K_M) skips any default-detection guesswork.

The three cases where Ollama wins

1. Developer machine — laptop or desktop. One curl and one ollama pull. The daemon idles at ~80 MB RAM until a model is loaded, then unloads after 5 min of inactivity. No systemd unit, no Python venv. The laptop stays a laptop; close the lid and Ollama waits on disk.

2. Hobby server — $7/mo KVM VPS or home NUC. systemd-managed out of the box (the install script registers ollama.service), automatic crash restart, on-disk model cache, OpenAI shim at :11434. Pull a 7B-8B model, point anything at http://box:11434/v1, walk away. 4-8 tokens/sec on a $7/mo VPS; 80+ on a box with one L4 or 3060.

3. Mac mini (M-series). Apple Silicon is the rare case where CPU inference is genuinely usable: unified memory lets a qwen2.5-coder:32b Q4_K_M at ~22 GB fit in the 32 GB Mac mini M4 Pro’s RAM and use the GPU and Neural Engine automatically. Metal is built in — no driver install. Steady-state 5-15 tokens/sec for 8B, 5-10 for 32B, 2-4 for 70B (which only fits on the 64 GB Mac Studio). For drafts and cron that can wait, the Mac mini is the cheapest LLM box you can buy.

The three cases where Ollama loses

1. Production — anything with a real SLA. Ollama is a daemon, not a serving framework. No request batching, no per-request KV-cache isolation, no streaming prioritization, no autoscaling. Under concurrent load from more than a handful of clients, the first one wins and everyone else queues. vLLM (vllm serve) batches and pushes 5-10x the throughput of Ollama on the same hardware.

2. Multi-tenant — shared GPU, multiple users. Ollama loads one model at a time per GPU. Two users wanting qwen2.5:32b and llama3.1:70b on the same box pay 22 GB + 48 GB = 70 GB of VRAM, which doesn’t exist on most cards. vLLM, SGLang, and TensorRT-LLM run multiple models concurrently with proper VRAM partitioning.

3. GPU-heavy — frontier inference, large context, tool loops. Ollama’s backend is llama.cpp — a single-request engine optimized for CPU and modest VRAM. It does not use FlashAttention-2 at full power, does not do speculative decoding, does not exploit tensor parallelism across GPUs. A 70B on two H100s with vLLM runs ~150 tokens/sec; with Ollama on two H100s, the same model runs at the speed of one H100 — the second card sits idle. For >70B, contexts >32k, or latency-critical agent loops, vLLM wins.

The wire-up — Ollama as an OpenAI provider

Once Ollama is running, every OpenAI-compatible client works. Hermes Agent is one example:

hermes chat --setup local --base-url http://localhost:11434/v1 --model qwen2.5-coder:32b -q "Reply with OK"

Same flags work over LAN, over a Cloudflare Tunnel, or to a remote GPU box running Ollama on CUDA. Other clients against http://host:11434/v1: openai-python, litellm, langchain, Continue.dev, Cursor (custom OpenAI base).

What not to do

• Don’t pull a 70B on a 16 GB Mac — the daemon OOMs mid-prompt. 8B fits 16 GB; 32B Q4_K_M needs 24+ GB; 70B Q4_K_M needs 48 GB.
• Don’t leave the model cache on the root partition. Ollama defaults to /usr/share/ollama/.ollama/models. On a $7/mo KVM with a 20 GB root disk, the first 70B pull fills the disk — symlink to a data volume first.
• Don’t assume CPU is fast enough for chat. A qwen2.5-coder:32b at 2-5 tokens/sec on a CPU-only VPS is unusable for interactive chat. Fine for hourly cron; wrong for Slack bot replies.
• Don’t run multiple large models on one GPU. Ollama does not partition VRAM; two 32B models on one L4 (24 GB) does not fit. Unload with ollama rm.
• Don’t skip :instruct. llama3.1:8b (base) is a text-completer, not a chat model — use llama3.1:8b-instruct or llama3.1:8b-instruct-q4_K_M.
• Don’t confuse q4_K_M with q4_0. Both 4-bit; q4_K_M is the smarter K-quant. Default to q4_K_M.

Verification checklist

#	Command	Proves
1	systemctl status ollama	Daemon alive
2	ollama list	Models on disk
3	curl http://localhost:11434/v1/models	OpenAI shim reachable
4	ollama ps	Model loaded in RAM/VRAM
5	One chat call from any OpenAI client	End-to-end works

Sources

• Ollama — official site
• Ollama — GitHub
• Ollama — OpenAI compatibility
• Ollama — REST API + tag conventions
• ABS companion: Hermes Agent: Local OpenAI-Compatible Provider
• ABS companion: Hermes Agent on a Mac mini M4
• ABS companion: Hermes Agent on Your Local PC

Last verified: 2026-07-15 against Ollama 0.5.x. Validated via curl /v1/models on macOS and Linux hosts.