Can my Mac run AI?

Memory a model needs

Memory = weights + context memory + runtime. Weights are the real download size of the file you pick, read from Hugging Face. When a size is unknown we use parameters x bits per weight / 8, with typical averages per quant type (Q2_K 2.96, Q3_K_M 3.91, Q4_K_M 4.8, UD-Q4_K_XL 4.9, MXFP4 4.25, NVFP4 4.5, Q5_K_M 5.7, Q6_K 6.56, Q8_0 8.5, BF16 16). Runtime is 0.7 GB, or 2% of the weights when that is more, for compute buffers and allocator slack: measured buffers stay near that size whatever the model. Against our own Mac app's measured memory for six models, and Apple's published totals for two models at 65,536 tokens, this lands within 15%. Where someone measured the memory of the exact file, we use that run instead and say so. Checked 2026-10-19.

• raw.githubusercontent.com/ggml-org/llama.cpp/master/gguf-py/gguf/constants.py
• machinelearning.apple.com/research/mlx-distributed-inference
• pocketweb.tools/mac/best-local-llm-for-mac

Context memory (KV cache)

Each token of context stores keys and values: 2 x layers x KV heads x head size x 2 bytes (16‑bit cache, the default in llama.cpp and MLX), read from each model's config.json. Sliding-window layers keep at most their window (2,048 tokens on Muse Spark Open, for example), so they add a fixed amount instead of growing. MLX is confirmed to cap them on Muse Spark Open; whether llama.cpp does for that model is not verified, so its GGUF figures may run low at long context. Models with compressed attention (MLA) store one small latent per layer. Linear-attention layers keep a small fixed state, counted in the runtime share. In llama.cpp you can store the cache at lower precision with --cache-type-k and --cache-type-v (-ctk and -ctv): per element that is q8_0 1.0625 bytes, q4_0 0.5625 bytes, from the size of each type's data blocks. A lower-precision V cache needs flash attention, which llama.cpp turns on by itself unless you switch it off. Measured runs are taken to use the 16‑bit default. Checked 2026-10-19.

• huggingface.co/unsloth/Muse-Spark-Open-816B-A42B/raw/main/config.json
• raw.githubusercontent.com/ml-explore/mlx-lm/v0.32.2/mlx_lm/models/muse_spark.py
• huggingface.co/meta-models/Muse-Glimmer-30B/raw/main/config.json
• github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp
• raw.githubusercontent.com/ggml-org/llama.cpp/master/gguf-py/gguf/constants.py

How much memory a Mac's GPU can use

macOS only lets the GPU use part of a Mac's memory. Apple publishes no formula, so we read the default from public llama.cpp, Ollama and MLX logs where one exists. On macOS 26 and later: 8 GB: 2/3, about 5.7 GB (medium evidence); 16 GB: 0.74, about 12.7 GB (strong evidence); 18 GB: 0.74, about 14.3 GB (medium evidence); 24 GB: 0.74, about 19.1 GB (strong evidence); 32 GB: 0.78, about 26.8 GB (strong evidence); 36 GB: 0.78, about 30.2 GB (medium evidence); 48 GB: 0.78, about 40.2 GB (strong evidence); 64 GB: 0.81, about 55.7 GB (strong evidence); 96 GB: 0.81, about 83.5 GB (weak evidence); 128 GB: 0.84, about 115.4 GB (strong evidence); 192 GB: 0.84, about 173.2 GB (weak evidence); 256 GB: 0.87, about 239.1 GB (weak evidence); 512 GB: 0.87 (assumed), about 478.3 GB (unconfirmed evidence). Why the weaker rows are weaker: 96 GB: One log, from 2026-09-30. 192 GB: One M2 Ultra log, from 2025-10-30. 256 GB: Two M3 Ultra logs, possibly from the same person. 512 GB: No confirmed default. Logs show 464 GiB (about 0.906), but the same value appears on a macOS 15 Mac, so it is probably a manual override. We use 0.87, the 256 GB ratio, which may understate a 512 GB Mac by about 20 GB. On macOS 15 or earlier it was 2/3 up to 32 GB and 3/4 above. The limit can be raised with sudo sysctl iogpu.wired_limit_mb=<MiB>; it resets on restart and macOS needs memory of its own, so leave headroom. The setting cannot go past a second limit, the kernel's cap on all locked memory, which Apple's open-source macOS kernel sets by memory size (16 GB: 13.1 GB; 32 GB: 27.1 GB; 64 GB: 56.3 GB; 128 GB: 116.8 GB; 256 GB: 241.9 GB; 512 GB: 515.4 GB) and two measured Macs match exactly. That cap sits a little above the default at every size, so the default is what a model gets, and we suggest the setting only when a model fits under the cap. It may still fail. Checked 2026-10-19.

• github.com/geerlingguy/ai-benchmarks/issues/55
• github.com/ollama/ollama/issues/13460
• github.com/ollama/ollama/pull/14604
• github.com/waired-ai/waired-agent/issues/857
• github.com/ggml-org/llama.cpp/issues/20704
• github.com/AshiqIqbal1/Whisperlet/pull/81
• github.com/OpenCSGs/csglite/issues/73
• github.com/ggml-org/llama.cpp/pull/21048
• github.com/ollama/ollama/issues/15545
• github.com/ggml-org/llama.cpp/pull/16669
• github.com/vllm-project/vllm-metal/pull/764
• github.com/ggml-org/llama.cpp/pull/16634
• github.com/r0shi/harborclerk/issues/684
• github.com/ggml-org/whisper.cpp/pull/3822
• github.com/red-hat-ai-dev/vLLM-zero-to-hero-pt1/issues/7
• github.com/ggml-org/llama.cpp/issues/20141
• github.com/ggml-org/llama.cpp/issues/19679
• github.com/ggml-org/llama.cpp/issues/20011
• github.com/ggml-org/llama.cpp/issues/21013
• github.com/mdegans/drama_llama/issues/125
• github.com/ggml-org/llama.cpp/issues/16475
• github.com/ggml-org/llama.cpp/issues/17607
• github.com/evanwtf/local-llm/issues/30
• github.com/ollama/ollama/issues/12847
• github.com/ollama/ollama/issues/12900
• github.com/ggml-org/llama.cpp/issues/19335
• github.com/ggml-org/llama.cpp/issues/17291
• github.com/ggml-org/llama.cpp/issues/16093
• github.com/ggml-org/llama.cpp/discussions/2182
• github.com/ml-explore/mlx-lm
• github.com/waired-ai/waired-agent/issues/857
• github.com/apple-oss-distributions/xnu/blob/main/osfmk/vm/vm_kern.c
• github.com/apple-oss-distributions/xnu/blob/main/osfmk/vm/vm_page.h
• github.com/osaurus-ai/vmlx-swift/pull/608

Several Macs linked together

For a group of Macs with the same memory, we add up what each one's GPU can use: two 256 GB Macs on macOS 26 and later come to about 478.3 GB. Assumes the model splits evenly across Macs of the same memory size (as exo and MLX distributed do) and that linking them costs no memory on any one Mac. Neither is measured. We give no estimated speed for a group; only published runs, each with its number of Macs. Checked 2026-10-19.

• machinelearning.apple.com/research/mlx-distributed-inference

How much memory a PC can use

On a graphics card we take its video memory minus 0.5 GB for the display. Runtimes like llama.cpp, LM Studio and Ollama can put layers that do not fit into system memory; we allow up to 60% of it, and any model that needs it is graded Tight. Both numbers are our assumptions, not measurements. AMD documents up to 96 GB of GPU memory on a 128 GB Ryzen AI Max machine; on smaller ones we assume the same 75% share. NVIDIA publishes no GPU limit for DGX Spark, so we assume 84% of its memory, the share macOS gives a 128 GB Mac. Checked 2026-10-19.

• amd.com/en/blogs/2025/amd-ryzen-ai-max-395-processor-breakthrough-ai-.html
• nvidia.com/en-us/products/workstations/dgx-spark

Reply speed

Writing each token reads the model's active weights once, so speed follows memory bandwidth: seconds per token = active GB / (bandwidth x 0.85) + a fixed cost per token of 2 ms on Apple for standard models, 5 ms for hybrid linear-attention models (Qwen3.5 and 3.8), 6 ms for mixture of experts; 1.3 ms on NVIDIA and DGX Spark for standard models and 2 ms for mixture of experts. Active GB comes from the file's tensor table where we computed it, else from a similar file of the same model whose table we read, scaled by file size, else from the active parameters at the file's own bits per weight. When a PC spills into system memory, that part reads at 60 GB/s x 0.6 (assumed dual-channel DDR5). Fitted to 53 published llama.cpp and MLX runs, it had a median error of 6% and every point within 20%; that fit is in-sample. AMD and Intel cards borrow the NVIDIA values and are marked rough. Speeds are at short context; long chats run slower. Checked 2026-10-19.

• github.com/ggml-org/llama.cpp/discussions/4167
• github.com/ggml-org/llama.cpp/discussions/15396
• github.com/ggml-org/llama.cpp/discussions/15013
• github.com/ggml-org/llama.cpp/blob/master/benches/dgx-spark/dgx-spark.md
• github.com/ggml-org/llama.cpp/pull/27390
• machinelearning.apple.com/research/mlx-distributed-inference
• github.com/john-rocky/apple-silicon-llm-bench

Why speeds are a range

The top of the range is the formula above. The bottom is 65% of it on M1 and M2 chips, where our own Mac app ran about a third under the formula on two mid-size models, and 85% everywhere else. The M1 and M2 rule rests on those two points only. Checked 2026-10-19.

• github.com/ggml-org/llama.cpp/discussions/4167
• pocketweb.tools/mac/best-local-llm-for-mac

When we do not estimate speed

We show no estimated speed for weights under 2 bits each, which are limited by computation rather than bandwidth (a 1‑bit 27B model ran at about a quarter of the bandwidth limit), or for mixture-of-experts models above 120B total parameters, where the formula ran 40% fast on Muse Spark Open against a measured 18 tokens per second. Those rows show measured speeds where someone published them, and "not measured" otherwise, and a model with no speed figure is graded Tight at best. Checked 2026-10-19.

• machinelearning.apple.com/research/mlx-distributed-inference
• huggingface.co/blog/meta-muse-spark-open

Measured beats estimated

When someone published a speed for the same model, quant and chip, we show that number, labelled measured, with its source and caveats, instead of the formula. A run on another runtime (Meta's ExecuTorch numbers for a GGUF, for example) or one that does not say which GPU-core version of a chip it used is shown for comparison only. Our own measurements come from PocketWebTools for Mac on a 16 GB M1 Pro and a 32 GB Mac mini M4. A speed measured at a context more than 4 times shorter than the one you picked (or at an unstated one, which usually means a short test) is labelled measured at short context, since long chats run slower. Measurements of several Macs linked together never stand in for one Mac, and each is shown with its number of Macs. Model data checked 2026-10-19.

• pocketweb.tools/mac/best-local-llm-for-mac
• machinelearning.apple.com/research/mlx-distributed-inference

Grades

Won't fit: the model needs more than the GPU can use (plus the allowed system memory on a PC). Tight: it fits but needs more than 90% of that memory, spills into system memory, replies under 8 tokens per second, or has no speed figure for that machine. Runs well: everything else. Every grade states the quant, the context length (8,192 tokens unless you change it) and the memory figure it assumed. Speech, voice and image models are graded on memory only. Checked 2026-10-19.

Time to the first word

Reading your prompt is limited by computation, not bandwidth. We scale each chip's published llama.cpp prompt speed on a 7B model by 6.6B / the model's active parameters, x 0.75 for mixture of experts. It matched measured runs within 8% for standard models. Checked 2026-10-19.

• github.com/ggml-org/llama.cpp/discussions/4167
• github.com/ggml-org/llama.cpp/discussions/15013
• github.com/ggml-org/llama.cpp/discussions/15396