Local AI: what your machine actually runs

The essentials

Two questions, in this order

They are not interchangeable. The first one is binary: it fits or it does not. Only if the answer is “it fits” does the second question exist at all.

The calculation that settles 90% of the doubts

theoretical tokens/s = bandwidth (GB/s) ÷ GB read per token
tokens/s in practice ≈ 75% of that number

An example that comes up a lot: the DGX Spark has 128 GB of memory and 273 GB/s of bandwidth. A compressed 70-billion-parameter model weighs about 40 GB. So 273 ÷ 40 ≈ 6.8 tokens/s at the ceiling, and about 4.5 in practice. It fits with room to spare — and it is slower than you read.

On the other side, the RTX 5090 has 1792 GB/s, the fastest memory in consumer hardware. But it has 32 GB. That same 40 GB model does not go in. The fastest conveyor belt on the market, standing still, because the box does not fit on it.

Two expensive machines, each answering a different question well. That is why “which is the best card for AI” is a badly formed question.

Before anything else

What do you want it for?

The use decides the model size, and the size decides the machine. Most people choose in the opposite order — they buy the card, then look for what it runs. That's where the money goes wrong.

Task, and the model size it really takes

The four reasons people run local

Choose in this order: use → model size → machine. The people who regret the purchase are the ones who went the other way.

Before you spend money

The 8 mistakes that make someone buy the wrong machine

They all come from the same root: looking at one number only, or looking at the number the store chose to show you.

Quick diagnosis

What actually runs in each memory tier

Assuming ~Q4 compression (the standard for running at home), a normal-sized conversation, and subtracting what the operating system already occupies. If all you want to know is “is my card good enough”, this is the table.

“Runs well” = fits with context headroom and delivers usable speed for that class of card. Different bandwidths at the same capacity change the experience — use the calculator below for your case.

Stacking cards

Two cards, three cards — does it get faster?

It's the first thing people think of when they read that a 70B doesn't fit. The answer is worth knowing before you spend: stacking cards buys you room, not speed.

What actually happens

In the mode everyone actually uses (layer-split, the default in llama.cpp, Ollama and LM Studio), the model is sliced by layers across the cards. To produce one token, card 1 runs its layers, hands the result to card 2, and so on — one at a time. Add it up and three identical cards behave exactly like one card of that bandwidth with three times the memory.

3 cards × (one third of the weights ÷ 1008 GB/s) = all the weights ÷ 1008 GB/s

So the memory adds up and the speed does not. It's the same lesson as the Mac cluster that dropped from 117 to 29 tok/s — this time inside a single case.

A real setup

Three RTX 4090s (24 GB, 1008 GB/s each) on a mining motherboard with PCIe 4x slots, running a 70B: 10–14 tok/s measured. If bandwidth added up, that would be around 57. One card's bandwidth against a 70B at Q5/Q6 predicts 13–16. The measurement lands on the prediction.

Why the slow slots don't ruin it

In layer-split, what crosses the bus between cards is the activation tensor at the seam — a few MB per token, not the weights. That's why a mining board with 4x slots works fine for one conversation at a time. PCIe matters for loading the model and for tensor-parallel mode. Practical consequence: don't spend on a top-end motherboard with multiple x16 slots to run local AI at home. That money buys more VRAM.

When bandwidth really does add up

In tensor-parallel (vLLM, or --split-mode row in llama.cpp), every layer is divided across the cards and they work together. Effective bandwidth then approaches the sum — but it needs a fast interconnect (NVLink, or at least PCIe x8/x16), because the cards synchronise at every layer. On x4 mining risers, tensor-parallel ends up worse than layer-split. Serving many users at once also exploits several cards well; this page is about one conversation at a time.

The bill nobody mentions

Three RTX 5090s draw about 1725 W on their own; with the rest of the machine, roughly 1925 W. A 127 V / 15 A circuit tops out at 1905 W and should not be run continuously above ~1524 W. Three 5090s do not fit on an ordinary household circuit of that kind — it takes a dedicated line, a 220 V circuit, or fewer cards. Three 4090s (~1550 W total) are already over the continuous limit. Add the heat: that's a small oven running in the room.

What the build actually needs

• Don't mix card models. In layer-split the slowest one drags the whole chain, in proportion to how much of the model it holds. Four identical cards beat two fast ones plus two slow ones. That applies across generations of the same brand too.
• Motherboard: count the slots before you count the cards. You need one PCIe slot per card. Ordinary desktop boards have one or two usable ones. Dual-socket Xeon server boards, common on the used market, reach four or more — typically two at 16x and the rest at 1x — and take 64 to 512 GB of cheap RAM.
• A 1x slot works, but it charges you at load time. Once the model is in memory the bus is barely used: only the activation tensor crosses it, a few MB per token. But loading 40 GB through a ~1 GB/s slot takes minutes. Once loaded, the speed is the same as on a 16x. A narrow bus costs patience up front, not performance afterwards.
• System RAM: 16 GB is the floor. It is used to load the model before handing it to the cards, and for whatever runs on the CPU. With many cards, 32 GB is more comfortable.
• Operating system: Linux is the normal route for a multi-card build, and Ubuntu is the best documented.

The short version

• Want to fit a bigger model? Stacking works, and it's often the cheapest way to reach 48 or 72 GB.
• Want more speed on a model that already fits? Stacking does nothing. Buy one faster card instead.
• Mixing different cards? The slow one drags the whole chain, proportional to how much of the model it holds.
• Check the wall socket and the PSU before the graphics card. It's the constraint people discover last.

One honest caveat: in practice you lose a little VRAM per card (driver context, buffers) and a little speed at the seam between layers. The calculation above is the ceiling.

Idle hardware

Got a mining rig sitting in the garage?

When Ethereum left proof-of-work, a lot of people were left with several identical cards, a serious power supply and a motherboard with plenty of slots. That is a usable shape — for one kind of model, and not the one you would guess.

The finding, and it is counter-intuitive

A rig has a lot of memory added up and modest bandwidth. That is bad for a dense model and excellent for MoE, because MoE occupies a lot of memory and reads little per token. It is exactly the shape of the hardware. Run through this page's own formula, on 6 × RTX 3060 12G (72 GB, 360 GB/s):

Note what does not fit: a 120B MoE asks about 73 GB of weights, and on top of that go the context and 1.5 GB of runtime per card. On six 12 GB cards that overflows the 72 GB. The rig runs MoE well — up to the size its memory actually holds.

Required caveat: the formula overestimates MoE — routing between experts charges its own price. The real number lands well below. The qualitative conclusion holds with room to spare: if you have a rig, look for MoE.

Which rig cards are worth reusing

Rule: below 8 GB per card it barely pays. An 8B with context already does not fit in 6 GB.

The electricity bill, which is what kills it

A rig of 6 × RTX 3060 12G pulls about 1170 W. Running 24 hours a day that is roughly 842 kWh a month — the practitioner who reported this pays about 1 real per kWh, so around R$ 850. Whoever used to mine knows this bill. The difference is that now there is no coin reward to offset it. Switch the rig on when you are going to use it, don't leave it standing all day. For a mid-sized model, one or two cards do the job at a fraction of that.

The Xeon option: cheap memory instead of cards

A dual-Xeon server board with 8 channels of DDR4 delivers around 150 GB/s — comparable to a modern mini PC — and takes 128 to 512 GB of cheap RAM. Same lesson: MoE turns wide, cheap hardware into a usable machine. It does not rescue a large dense model.

Side by side

The same model on every machine

Change the model in the calculator above and this whole table rebuilds. Sorted by what fits, fastest to slowest. Machines that do not fit show the speed they would have — struck through, because it is hypothetical.

Which model to download

Catalogue of local models

Knowing you need “an 8B” still doesn't tell you which file to download. Click a row to load the model into the calculator, or go straight to Hugging Face to fetch the file.

In one sentence: this table is the shortcut; Hugging Face is the archive.

How to read a model page and bring it here

When you open a model on Hugging Face, look for the Files tab. That's where the files you actually download live. Three things matter:

• The name carries the size. Llama-3.1-8B is an 8-billion-parameter model. That's the number the calculator wants in “parameters” mode.
• The suffix is the compression. Q4_K_M, IQ4_XS, Q8_0. The number after the Q is roughly how many bits per weight. Q4 is the home-use default.
• The GB next to the file is not the memory you'll spend. It's just the download size. Context is missing — that's why the calculator adds it separately.

If you have the file in front of you and only know its size in GB, use the “I know the file size” mode in the calculator: it converts to parameters for you, using the compression you picked.

A warning for beginners: some official models (Meta's, for example) are gated — Hugging Face asks for an account and a licence acceptance before releasing the download. Nothing is broken; that's the licence. Community GGUF conversions are usually open, and that's what the search links below surface first.

Q4 GB is the typical size of the downloaded file — not a benchmark and not the memory in use. On this machine applies the page's formula with the memory, bandwidth and context you picked in the calculator above: change the hardware up there and this whole column rebuilds. Every row uses the same token budget; what changes is the memory it costs, which scales with the model’s size. This is not llama-bench.

Read the MoE numbers with a discount. The formula assumes only the active parameters are read, and that gives an over-optimistic ceiling: routing between experts costs something. A case from our own measured table: gpt-oss 120B was measured at 70–80 tok/s on a Mac Studio, well below what the plain division suggests. For MoE, treat the column as “it'll be fast”, not as the number.

Reality

Numbers other people measured

The calculator gives you the estimate. These are published third-party bench results — useful to calibrate expectations and to see the effect of the software used. Filter for what matters to you.

measured published bench · estimate formula applied, not a measurement. All of them generating a response (decode), one conversation at a time.

What this table teaches that the formula does not

• The software matters almost as much as the card. Same model, same Mac: 26–28 tok/s on one runtime, 17.8 on another, 5–7 on a third with factory defaults on. Before you change hardware, change software.
• MoE changes the game in the middle tier. A 30B MoE with 3B active runs 30–40 tok/s on an iGPU where a dense 27B drags at 5–8. Same memory occupied, six times the speed.
• Extra capacity is not extra speed. The RTX PRO 6000 has 96 GB against the 5090's 32 GB — and exactly the same bandwidth. It lets you run what did not fit; it does not let you run what already fitted any faster.
• A small model does not exploit a big card. A 1B model reached 281 tok/s. You will not feel the difference between 200 and 280 — at that size, any modern card does the job.

Step by step

How to find your two numbers

Nothing to install. It takes a minute.

Reference table: bandwidth per card

Found your card's name in Task Manager? Its bandwidth is here. Figures from the manufacturers' official specs.

Recipes

Five builds, with honest numbers

Every recipe says what you get and what you give up. The tokens per second come from this page's own formula, at 8k of context — they are estimates, not benchmarks. Watts are for the whole machine: cards plus 150 W for the rest.

Decision

Buying guide by goal

There is no “best”. There is the right one for what you are going to do. Every recommendation below comes with what you give up.

Vocabulary

Glossary without jargon

The terms you will meet in any tutorial, in plain English.

Token

A fragment of a word. In English, a common word is usually 1 to 2 tokens. “Tokens per second” is, in practice, the speed at which text appears on screen.

Parameters (the “8B”, “70B”)

How many billions of numbers the model has. It is the “factory” size and it is not the size of the file you download — the file depends on compression.

Quantisation (Q4, Q8, IQ4_XS)

Compressing the weights, storing each number with fewer decimals. Q4 uses about 4 bits per weight, Q8 uses 8. Fewer bits: the model takes less space, runs faster and loses a little quality. Q4 is the standard for running at home — the loss is small and the gain is enormous. Variants such as IQ4_XS squeeze a bit further and are sometimes the difference between fitting and not fitting.

VRAM and unified memory

VRAM is the memory that belongs to the graphics card — fast and not expandable. Unified memory (Macs, some mini PCs) is a single pot shared between processor and GPU: more flexible, almost always with less bandwidth than a dedicated card.

Memory bandwidth (GB/s)

How many gigabytes memory delivers per second. It is the number that decides generation speed. It rarely appears in the listing; it is always in the manufacturer's spec sheet.

Context and KV cache

Context is how much the model “remembers” of the conversation and the open files. The KV cache is the memory that takes up — and it is not included in the model file size. It is why a “6.6 GB” model actually occupies 17.9 GB.

Dense vs MoE (Mixture of Experts)

A dense model reads all of its weights to generate every token. A MoE is split into experts and activates only a few at a time: it needs all the memory to fit, but reads little per token. That is why a “30B-A3B” (30 billion total, 3 active) occupies like a big model and runs like a small one.

Offload

When the model does not fit entirely on the card, the software leaves some of the layers in system memory and processes them on the processor. It works — and it is far slower, because that part loses the card’s bandwidth and comes to depend on the RAM and the CPU instead. It is there to run what would not fit; it is not there to run fast.

Prefill and decode

Prefill is reading your question — it depends on the GPU's compute power. Decode is writing the answer — it depends on memory bandwidth. The calculator on this page estimates decode, which is the part you feel while the text appears.

GGUF, Ollama, LM Studio, llama.cpp, MLX

GGUF is the most common file format for compressed models. llama.cpp is the engine that runs that format; Ollama and LM Studio are friendlier ways to use the same engine (LM Studio has a graphical interface and is the easiest way to start). MLX is Apple's engine, tuned for M-series chips — on a Mac it is usually much faster than the alternatives.

NPU and TOPS

An NPU is a chip dedicated to AI tasks, measured in TOPS. It is great for audio transcription and computer vision. It does not solve the bottleneck of a large LLM, which is memory bandwidth. High TOPS in the marketing does not mean high tokens per second.

Transparency

Sources, method and limits

Where the numbers come from

StudyMarkets built this guide on top of third-party public benches and explanations. The core formula (bandwidth ÷ size, practice ≈ 75%), the 6 vs 30 tokens/s ruler and the memory-in-use examples come mainly from the ViktorKav channel. The measured tokens per second come from the channels and sites listed below. Card bandwidth figures come from the manufacturers' official documentation.

• ViktorKav (@VKav) — formula, ruler and examples. Apple took local AI from NVIDIA · The right local AI for 4, 8 and 16 GB · channel
• Alex Ziskind (@AZisk) — 5090 vs 5080 vs laptop vs Mac, Mac mini cluster. channel
• Jeff Geerling — Mac Studio clustered over Thunderbolt 5. channel · write-up
• NetworkChuck — 5 Mac Studios on a network: the drop from 117 to 29 tokens/s. channel
• ttio2tech — 3090 vs 5090 on the same 27B model.
• MyAIHardware — llama.cpp table on 5090 / 4090 / 3090 / Mac. benchmarks
• Unsloth — sizes and quality of GGUF compressions. Dynamic 3.0 GGUFs
• HWCooling and Phoronix — integrated graphics and the bandwidth bottleneck. HWCooling · Phoronix

What this page is not

• It is not a benchmark. It is an order-of-magnitude estimate for deciding a purchase. Real numbers depend on the software, the version, the context and the configuration.
• The 75% rule is optimistic on small models. On an 8-billion model, observed real efficiency lands closer to 55% of the ceiling; on large models the rule holds up well.
• It only covers generating the answer. Reading your prompt depends on the GPU's compute power and is not part of this calculation.
• None of the channels cited endorses this tool. The measurements are theirs; the calculation and the text are ours.
• If your machine's official spec sheet disagrees with a preset, trust the spec sheet and use Custom mode.

How to cite

StudyMarkets. Local AI: what your machine actually runs, 2026. Generation formula: memory bandwidth ÷ size read per token (ViktorKav, “Apple took local AI from NVIDIA”, YouTube). Tokens per second measured by the channels and sites cited on the page.