Guides

New to local LLMs? Start here. What "local" means, what hardware you need, and how to pick a first model.

Three ways to get a model running on your machine: Ollama, LM Studio, llama.cpp. Walks through install + first chat.

Use this when you already have a GPU and want to know which models fit.

The simple formula: parameters x bytes per param x overhead. Worked through Q4, Q5, Q8, and FP16.

My picks by budget and use case. Real tokens-per-second numbers where I have them.

RTX 3090, 4090, 7900 XTX, A5000. Which makes sense in 2026.

Under $500 picks. RTX 3060 12GB still leads on $/GB VRAM.

What it actually runs, what it does not, and how thermals behave in a real room.

The used 3090 is the best $/GB you can buy. Caveats included.

Which tier is the right break-point for you. Not always the 4090.

Bandwidth still favors the 5090. October 2026 Amazon asks are a different question from the May price.

Default buy is the GMKtec EVO-X2 128GB at ~256 GB/s. The Beelink GTR9 Pro is the same silicon with a ~$700 premium and an Intel E610 NIC caveat. SER9/SER10 at ~90 GB/s are hobby 7–14B boxes.

October 2026 Amazon buy-now: Mac Studio M5 Max 36GB, Mac mini M5 Pro and M4 Pro 24GB, plus the Strix Halo EVO-X2. 64GB+ configures at Apple.

When a Mac wins, when a PC wins. Different workloads land in different places.

Every command I actually use, with the flags I always forget.

Pick the right tool for the job. They are not interchangeable.

Native Windows + WSL2 routes. Driver gotchas included.

Ollama, LM Studio, and llama.cpp on Apple Silicon. What works best for which model size.

Why Q4 fits where FP16 cannot, and where quality actually drops.

The two formats you keep seeing. Which to use when.

CPU, RAM, storage, and PSU notes most guides skip.

What 8B, 70B, and 405B need. With Q4/Q8/FP16 breakdowns.

Original Qwen3 sizes, from 0.6B through the big MoE. VRAM, not a leaderboard.

What 16, 24, and 32 GB actually hold from the May–October 2026 local models. Announcements without weights stay off the list.

Liquid's 8.3B MoE with 1.5B active. Q4 fits an 8 GB GPU; the license is lfm1.0.

DeepSeek R1 and V3, including realistic options for the 671B beast.

Microsoft Phi-4 14B is the easiest mid-size model to run well.

Google Gemma 3, including the 27B that fits in 24GB VRAM at Q4.

Mistral 7B, Mixtral 8x7B, and Mistral Small / Large.

Three cloud always-on agents, then one local blueprint and the full memory legend.

Muse stays on a Secure VM. The local path is Ollama, Open WebUI, and a model that fits.

Meta’s open 30B agent model. Their 17 GB K-quant next to the site formula.

Two words, cloud computer, no local Grok weights. OpenBot or OpenHands if the box is yours.

Lowercase dots, GPT-6 Astra, paid plans. Operator is not the current product.

CopilotKit’s self-hosted agents, kept apart from the healthcare RPA company.

Folder handoff in the cloud. Locally: OpenHands, browser-use, and approval on.

Formerly Background Agents. Aider, OpenHands, and Continue on hardware you own.

Agent Canvas and the CLI. 7B is a demo. 27–32B is the coding tier.

Mac mini 24 GB, Studio 36 GB, and the 128 GB EVO-X2, mapped to model tiers.

Worked 7B through 70B examples, plus a scaffold pad that is not a vendor spec.

Comet keeps their index. browser-use keeps the click-path on your machine.

The team product, not dots. Size one shared model for the group.

Microsoft Graph stays with Microsoft. Local RAG is the documents you actually stored.

The homepage was too thin to quote. OpenHands is the sandbox I can describe.

Cloud VMs plus a Desktop. One shared coding model beats a GPU per role.

Terminal, git, Ollama. 7B for a small edit, 24 GB for a 27–32B.

Text extraction on 12 GB. Screenshots want 24 GB and a multimodal model.

Roles share a server. Three copies of a 7B is the expensive way to get a weak crew.

The graph is cheap. The model inside the node is the bill.

Proprietary desktop, local server, Bionic on the current homepage. Not open source.

A terminal agent with a sandbox. No unattended root recommendation.

Still a local IDE client. Not the platform I would build a team on.

Skills replace Gems, with Google’s dates. A modelfile is the local copy of a saved prompt.

The shortlist that actually holds up across coding, writing, and reasoning tasks.

Which open model gets closest to GPT-4-class code completion.

For drafting, editing, and tone control. Not the same as coding picks.

Models that handle tabular reasoning and structured outputs well.