A useful local setup has four parts: a model suited to your task, software that can run it, enough memory for your conversation, and a way to check the result. Start with one task and one runner. Add other tools when the work needs them.
Start with the job
| Your work | A starting recipe | How to check it |
|---|---|---|
| Private writing and summaries | Qwen3 14B, 4-bit, Ollama, 8K–16K context | Verify facts, quotes, and missing details. |
| Code explanations and refactors | Qwen3-Coder 30B, 4-bit, Ollama and an editor integration, 16K context | Inspect changes and run tests. |
| Questions about screenshots | Gemma 3 12B, 4-bit, a runner with image input, 8K context | Check text, counts, and numerical values against the image. |
These are editorial starting points from our shortlist, not claims that these are the newest or best models. Use the finder to see hardware choices and change the context size.
Ollama, LM Studio, MLX: which layer is which?
Ollama downloads and runs models, with a command-line chat and local API. Start here if you already use it. A text model can answer about pasted code, but it needs an editor integration to see files or participate in a coding workflow.
LM Studio combines a desktop interface, model management, chat, and local serving. It is an alternative starting app. Document features still depend on parsing the files and getting the relevant text into context.
MLX-LM runs text models through Apple’s MLX framework. It is useful for Apple silicon and Python workflows. Choose MLX-compatible weights: a GGUF file used by Ollama is a different artifact. Two versions labeled “4-bit” can differ in size, quality, and performance. Recheck the actual allocation when switching.
llama.cpp gives you direct control over a GGUF runtime, including GPU offload and benchmarking. It is a useful next step when you need to investigate allocation or reproduce a performance result.
Why can’t every AI app run every model file?
A downloaded model still needs software that understands its file format and a computer that can hold it in memory. Think of a video: the file, a player that understands it, a screen, and the device all play different roles. For local AI, the pieces are a model file, a compatible runner, a chat or editor interface, and your hardware. An app such as Ollama or LM Studio can combine the runner and interface.
GGUF files are commonly run with llama.cpp-based tools such as Ollama. MLX-LM usually uses weights converted for MLX. File names and bit depth alone do not establish compatibility. Hugging Face added a path for some GGUF checkpoints in Transformers on Apple silicon in September 2026, but its initial fast path covers only supported architectures and software versions. If a model will not load, check the file format, model architecture, runner version, and available memory before buying new hardware.
Add the tools your task needs
For coding, Continue can connect to Ollama inside your editor. Configure a supported model for the role: chat, autocomplete, and tool-driven agents have different requirements. A model that writes good functions does not automatically make a reliable autonomous agent.
For a document collection, you may need text extraction, chunking, retrieval, and an embedding model. A chat model alone does not index your files. Additional models and concurrent requests also need memory beyond the single-model estimate in our finder.
Choose context before buying memory
The context budget includes instructions, history, supplied material, and the reply. An 8K window is a useful starting point for short work; 16K or 32K gives more room. Full repository agents and very large documents may require more than this finder’s 4K–32K planning range. A model’s advertised maximum is not proof that your chosen machine can run it comfortably at that length.
On a 64 GB M1 Max, our everyday 8K recipe starts with Qwen3 14B while leaving space for other apps. Dedicated use allows a larger general model such as Qwen3 32B. These are memory-based starting points, not measured speed claims.
Read hardware estimates carefully
Specific desktop cards matter: the finder distinguishes RTX 5070, 5070 Ti, 4090, and 5090, plus two-card setups. RTX is one supported route; Apple silicon, DGX Spark, and other accelerators use different memory and software arrangements. GPU system RAM and dedicated VRAM are separate.
Ollama may spread a model across GPUs when it cannot fit on one. Total VRAM is only a starting check. Each card must hold its assigned tensors and buffers, and the machine needs suitable PCIe connectivity, physical space, cooling, and power. Do not assume two cards give twice the generation speed.
Compare complete-system cost with complete-system cost. Our budgets show their assumptions and exclude tax and peripherals. Power scenarios are separate from card ratings and power-supply capacity. For your own electricity estimate, measure watts at the wall and enter your active hours and local rate.
Local or frontier?
Local inference offers offline use after downloading and control over where the model runs. A frontier service may offer tools, modalities, or capabilities that your local setup lacks. Test both on the same representative tasks. Compare correctness, editing effort, privacy requirements, response time, and total cost; a parameter count cannot tell you which setup will finish your work.