Toolkit
← All posts

Running AI on your own hardware: local LLMs explained

Toolkit Blog · · Hardware & local AI · 9 min read

Every big AI chatbot runs in someone else's data center — but a quieter movement has been putting capable language models on ordinary laptops and desktops, where they answer to no one but you. This piece explains what that actually takes: what the model sizes mean, why quantization is the trick that makes any of it possible, which hardware spec genuinely matters, and what you can realistically expect compared to the frontier models in the cloud.

Why run a model locally at all

Let's be honest about the baseline first: the strongest models you can use today are cloud models, and nothing you run at home will match them. If you want maximum capability and don't mind the terms of service, a cloud subscription is the rational choice. People run models locally anyway, for four reasons that hold up.

Privacy. A local model processes your prompt on your own machine. Nothing is uploaded, logged, retained for "service improvement," or subpoenaable from a third party. For medical questions, contracts, journals, unreleased code — anything you'd hesitate to paste into a web form — that property is not a nice-to-have, it's the whole point. (It's the same architecture argument we make for this site's tools in why Toolkit runs in your browser: a file that never leaves can't leak.)

Cost structure. Local inference costs you hardware you may already own plus electricity. There's no per-token meter, no monthly cap, no plan tier. For heavy, repetitive workloads — bulk summarization, classification, code review over a private repo — "free after setup" compounds quickly.

Offline and always-on. A local model works on a plane, in a dead zone, and during a provider outage. It also can't be deprecated out from under you: a model file on your disk behaves the same way in five years.

Tinkering. You can swap models, adjust sampling, fine-tune on your own data, and wire the model into scripts with no rate limits. For a lot of people this is the honest main reason, and there's nothing wrong with that.

The counterweights: setup has real friction, output quality is a tier below the frontier, and your hardware — not a data center — sets the speed. The rest of this piece is about making those trade-offs concrete.

What "7B" and "70B" actually mean

Model names come with a number: 7B, 27B, 70B. The B is billions of parameters — the learned numerical weights inside the neural network. Parameters are where the model's knowledge and ability live, so more of them generally means a more capable model. They're also, quite literally, the thing you have to store and move through your processor for every generated word, which is why the parameter count is the single best predictor of what hardware you'll need.

One wrinkle has become essential to understand: many current models are mixture-of-experts (MoE) designs, which means only a fraction of the parameters do work on any given token. You'll see this written as two numbers. Alibaba's Qwen3.6-35B-A3B has 35 billion parameters in total but only about 3 billion active per token[5]; OpenAI's gpt-oss-20b has 21 billion total and 3.6 billion active[4]. The practical effect: you still need enough memory to hold all the parameters, but generation speed behaves more like a small model's. It's how mid-2026 open models manage to be both large and fast on modest hardware.

Quantization: how big models fit on small machines

A parameter is natively stored as a 16-bit number — two bytes. Straight arithmetic says a 7B model needs about 14 GB just for its weights, and a 70B model about 140 GB. By that math, almost nobody could run anything interesting.

Quantization is the fix: store each parameter with fewer bits. The open-source inference engine llama.cpp — the foundation under most local-AI tooling — supports 1.5-bit through 8-bit integer quantization "for faster inference and reduced memory use"[1]. At 4 bits per parameter, that 7B model shrinks to roughly 3.5 GB and a 27B model to under 14 GB — laptop territory. The compressed models are distributed as GGUF files, the format llama.cpp and its descendants require[1].

The catch is that quantization is lossy: you're rounding the model's weights, and below about 4 bits the quality loss becomes noticeable. The community's working consensus is that a 4-to-5-bit quantization of a bigger model usually beats a full-precision smaller one — and model builders now design for this outright. OpenAI post-trained gpt-oss-20b with 4-bit (MXFP4) quantization of its MoE weights specifically so it would "run within 16GB of memory"[4], and even published its benchmark evals at that quantization. Fitting on consumer hardware has become a design target, not an afterthought.

The hardware that matters: memory, not raw compute

Here's the part that surprises people who assume AI needs an exotic GPU: for local inference, the binding constraint is how much fast memory you have, not how many teraflops. Every generated token requires reading essentially all of the model's (active) weights, so the model must sit in memory that the processor can read at very high speed — and the memory's bandwidth largely sets your tokens-per-second.

That gives you two viable paths:

Two budget notes. Context — the conversation and documents the model is currently holding in its head — consumes memory on top of the weights, and long contexts consume a lot; leave headroom rather than loading the largest model that technically fits. And models can run from ordinary CPU RAM with no GPU at all; it works, just several times slower, because standard system memory has a fraction of the bandwidth.

The on-ramp: Ollama, LM Studio, llama.cpp

Ten minutes of setup, honestly. Three tools cover nearly everyone, and all three are built on the same GGUF-model foundations:

  1. Ollama — the developer favorite. A small command-line tool for macOS, Windows, Linux and Docker: ollama run plus a model name downloads a sensibly quantized build and drops you into a chat, and it exposes a local REST API so scripts and editors can talk to the model. Its library tracks current open models — the July 2026 releases list Kimi, GLM, DeepSeek, gpt-oss, Qwen and Gemma support in the same breath[2].
  2. LM Studio — the graphical option, "free for home and work use"[3]. You browse models in a UI, see which fit your machine, download and chat — no terminal. It also runs Apple's MLX model format on Apple Silicon and can serve an OpenAI-compatible API[3], so apps written for the cloud API can point at your laptop instead.
  3. llama.cpp — the engine itself, whose stated goal is LLM inference "with minimal setup and state-of-the-art performance on a wide range of hardware"[1]. Compile it and you get maximum control and the newest features first. Most people don't need to start here; it's where you end up when you outgrow the other two.

Fun fact: you may have already run a neural network locally without installing anything. Toolkit's vocal remover executes Meta's Demucs source-separation model entirely in your browser — a one-time ~172 MB model download, then your GPU (via WebGPU) or CPU does the splitting. Same principle as a local LLM, smaller scale.

The open-weight landscape, as of July 2026

"Open-weight" means the trained model file is published for anyone to download and run — the weights are open even when the training data and recipe aren't. The current families worth knowing, as of July 2026:

The pattern across all of these: permissive licenses (Apache 2.0, MIT), mixture-of-experts architectures, and a clear split between genuinely local-sized models (roughly 4B–35B) and open-but-datacenter models wearing the same branding.

What to realistically expect

A well-chosen 12B–35B model on a 16–24 GB machine is genuinely useful in 2026: solid summarization, drafting, translation, code assistance, question-answering over your own documents, and structured extraction — private, free per query, and fast enough to feel conversational. What it is not is a frontier model. Expect a noticeable gap on long multi-step reasoning, obscure knowledge, and large codebases, and expect more confident-sounding mistakes; verify anything that matters.

When not to bother: if you need best-available quality on hard problems, use a cloud model. If your laptop has 8 GB of RAM and no GPU, small models will run but may disappoint — try before buying hardware. And don't buy a GPU for a workload you haven't measured; install Ollama or LM Studio first, run a 4-bit model sized to the machine you already own, and let actual usage tell you whether more memory is worth the money. The barrier to finding out is one download and ten minutes.

Sources

  1. ggml-org — llama.cpp repository (README), accessed July 2026
  2. Ollama — official repository and releases, accessed July 2026
  3. LM Studio — official site, accessed July 2026
  4. OpenAI — gpt-oss-20b model card (Hugging Face), accessed July 2026
  5. Qwen — Qwen3.6-35B-A3B model card (Hugging Face), accessed July 2026
  6. Google DeepMind — Gemma models, accessed July 2026
  7. DeepSeek — DeepSeek-V4-Flash model card (Hugging Face), accessed July 2026
  8. Mistral AI — Mistral Small 4 announcement, accessed July 2026
  9. Meta — Llama organization page (Hugging Face), accessed July 2026

Related: Why Toolkit runs entirely in your browser · How AI vocal removal works · Try the in-browser vocal remover