LLM VRAM calculator
A model needs room for three things: its weights (the quantized file, about 0.6 GB per billion parameters at Q4_K_M), its context memory, which grows with every token of context, and about 0.7 GB for the runtime. Llama 3.1 8B at Q4_K_M needs about 6.7 GB at 8,192 tokens but 22.8 GB at 131,072, so a card or Mac that fits the short chat can fail the long one, and a Mac's GPU can use only part of its memory (12.7 GB of a 16 GB Mac).
How much memory does this model need?
How to pick the model
The longest chat or document you will give it.
8-bit and 4-bit are llama.cpp settings.
Memory needed
Needs about 6.7 GB: the smallest Mac that holds it has 16 GB (the GPU can use 12.7 GB), and the smallest graphics card has 8 GB (7.5 GB free).
- Weights
- 4.9 GB
- Context memory8,192 tokens
- 1.1 GB
- Runtime
- 0.7 GB
Which memory sizes hold it
Fits means under 90% of what the GPU can use, Tight means the rest of the way to 100%. Memory only: pick your machine below for the reply speed.
macOS version
Macs
- 8 GB MacGPU can use 5.7 GB, 1.0 GB shortWon't fit
- 16 GB MacGPU can use 12.7 GBFits
- 18 GB MacGPU can use 14.3 GB (not in Apple's lineup today)Fits
- 24 GB MacGPU can use 19.1 GBFits
- 32 GB MacGPU can use 26.8 GBFits
- 36 GB MacGPU can use 30.2 GBFits
- 48 GB MacGPU can use 40.2 GBFits
- 64 GB MacGPU can use 55.7 GBFits
- 96 GB MacGPU can use 83.5 GB (weak evidence)Fits
- 128 GB MacGPU can use 115.4 GBFits
- 192 GB MacGPU can use 173.2 GB (weak evidence, not in Apple's lineup today)Fits
- 256 GB MacGPU can use 239.1 GB (weak evidence)Fits
- 512 GB MacGPU can use 478.3 GB (assumed, not in Apple's lineup today)Fits
Graphics cards
- 8 GB card7.5 GB freeFits
- 12 GB card11.5 GB freeFits
- 16 GB card15.5 GB freeFits
- 24 GB card23.5 GB freeFits
- 32 GB card31.5 GB freeFits
- 48 GB card47.5 GB freeFits
- 96 GB card95.5 GB freeFits
We leave 0.5 GB for the display. Spills to RAM: the card runs what it holds and system memory the rest, much more slowly. We allow 60% of 64 GB (38.4 GB); past that, Won't fit. All three are our assumptions.
Check it on your machine
Adds what your GPU can use, the reply speed (measured where someone measured it, estimated otherwise) and a verdict. Nothing leaves your browser.
Get the next tool first
New local-AI tools and updates by email. Unsubscribe anytime.
How the calculator works
- Memory needed = weights + context memory + runtime.
- Weights are the real download size of the file you pick, read from the model's Hugging Face file list. Only a custom model, which has no file, uses parameters x the quant type's typical bits per weight / 8.
- Context memory (the KV cache) = 2 x full-attention layers x KV heads x head size x 2 bytes x tokens, from each model's config.json. Sliding-window layers stop growing at their window, compressed attention (MLA) stores one small latent per layer, and linear-attention layers keep a small fixed state that the runtime share covers.
- Runtime is 0.7 GB, or 2% of the weights when that is more. Measured runtime buffers stay near 0.7 GB whatever the model, so the common rule of thumb of about 10% overstates large models; the 2% is a margin for big files.
- Where someone published the memory of the exact file at a known context, the calculator starts from that run and moves only the context memory, and says so.
llama.cpp can also store the cache at lower precision with -ctk q8_0 -ctv q8_0 (8‑bit) or -ctk q4_0 -ctv q4_0 (4‑bit). For Llama 3.1 8B at 131,072 tokens that brings the total from 22.8 GB to 14.7 GB at 8‑bit and 10.5 GB at 4‑bit. A lower-precision V cache needs flash attention, which llama.cpp turns on by itself unless you switch it off.
Same model, different answers
Ten worked examples
Context length, quant and machine each move the answer. Every row uses the same estimator as the calculator above; on cards, what does not fit runs from system memory.
| Model | Machine | Needs / GPU can use | Reply speed | Verdict |
|---|---|---|---|---|
| Llama 3.1 8BQ4_K_M, 8,192 tokens | GeForce RTX 4060, 8 GB | 6.7 GB of 7.5 GB | 40 to 47 tok/s | Runs well |
| Llama 3.1 8BQ4_K_M, 131,072 tokens | GeForce RTX 4090, 24 GB | 22.8 GB of 23.5 GB | 127 to 150 tok/s | Tight |
| Qwen3 30B‑A3BQ4_K_M, 32,768 tokens | GeForce RTX 4090, 24 GB | 22.5 GB of 23.5 GB | 206 to 243 tok/s | Tight |
| Gemma 3 27BQ4_K_M, 131,072 tokens | GeForce RTX 5090, 32 GB | 28.4 GB of 31.5 GB | 70 to 82 tok/s | Tight |
| Qwen3 32BQ4_K_M, 8,192 tokens | GeForce RTX 5090, 32 GB | 22.6 GB of 31.5 GB | 61 to 72 tok/s | Runs well |
| Qwen3 32BQ8_0, 8,192 tokens | GeForce RTX 5090, 32 GB | 37.7 GB of 31.5 GB6.2 GB in system memory | 4.9 to 5.7 tok/s | Tight |
| gpt-oss-20bMXFP4, 8,192 tokens | GeForce RTX 5060 Ti, 16 GB | 13.0 GB of 15.5 GB | 97 to 114 tok/s | Runs well |
| Qwen3.6 35B‑A3BUD-Q4_K_XL, 32,768 tokens | 32 GB M4 | 23.7 GB of 26.8 GB | 25 to 27 tok/s1 | Runs well |
| gpt-oss-120bMXFP4, 32,768 tokens | 128 GB M5 Max (40-core GPU) | 65.9 GB of 115.4 GB | 66 to 78 tok/s | Runs well |
| Llama 3.3 70BQ4_K_M, 8,192 tokens | GeForce RTX 4090, 24 GB | 46.1 GB of 23.5 GB22.6 GB in system memory | 1.4 to 1.7 tok/s | Tight |
On graphics cards we leave 0.5 GB for the display and let system memory hold what does not fit; both are our assumptions. Reply speed is in tokens per second at short context. Measured figures come from the published runs listed below; estimates come from memory bandwidth, as explained under How we estimate.
- 1. Measured: 25 to 27 tok/s, Qwen3.6 35B‑A3B UD-Q4_K_XL on 32 GB Mac mini M4 at 32,768 tokens. pocketweb.tools. PocketWebTools for Mac: llama.cpp on Metal, every layer on the GPU, lowest and highest reply over the test prompts.
Why context length changes the answer
| Model and file | 8,192 tokens | 32,768 tokens | 131,072 tokens |
|---|---|---|---|
| Llama 3.1 8B, Q4_K_M | 6.7 GB | 9.9 GB | 22.8 GB |
| Llama 3.3 70B, Q4_K_M | 46.1 GB | 54.1 GB | 86.3 GB |
| Gemma 3 27B, Q4_K_M | 18.4 GB | 20.4 GB | 28.4 GB |
| gpt-oss-120b, MXFP4 | 65.0 GB | 65.9 GB | 69.5 GB |
| Qwen3.6 35B‑A3B, UD-Q4_K_XL | 23.2 GB (from a measured run) | 23.7 GB (from a measured run) | 25.7 GB (from a measured run) |
Decimal GB, 16‑bit cache. The measured run is our own Mac app's, on a 32 GB Mac mini M4.
The model's design decides how fast it grows. Gemma 3 27B keeps 52 of its layers on a 1,024-token sliding window, so its context memory at 131,072 tokens is 11.2 GB rather than growing in step with every layer. gpt-oss-120b uses a 128-token window on 18 layers, so its total grows by only 4.5 GB from 8,192 to 131,072 tokens. That is why the calculator reads each model's config instead of applying one rule.
Your app may pick the context for you. llama.cpp's server defaults to the model's own longest context, and its fit option, on by default, lowers settings you did not set until the model fits in memory, but not below 4,096 tokens of context. Set the context yourself if you need a long one, and size the memory for it here.
Why the quant changes the answer
| File | Weights | Context memory | Total |
|---|---|---|---|
| Q4_K_M | 4.9 GB | 1.1 GB | 6.7 GB |
| Q8_0 | 8.5 GB | 1.1 GB | 10.3 GB |
| BF16 | 16.1 GB | 1.1 GB | 17.8 GB |
Across the 23 standard models in our data with a Q4_K_M file, the file is 0.58 to 0.69 GB per billion parameters (median 0.61), because each quant mix keeps some layers at higher precision. Q8_0 files run a median 1.06 GB per billion and 16‑bit files 1.99.
Mixture-of-experts models can break the pattern. gpt-oss-120b ships its experts in MXFP4, so its 63.4 GB file is the same size at 8‑bit (63.4 GB) and grows only to 65.4 GB at 16‑bit: the experts stay 4‑bit in all three.
GPU memory vs a Mac's unified memory
On a graphics card we leave 0.5 GB of video memory for the display, so a 24 GB card keeps about 23.5 GB for a model. llama.cpp, LM Studio and Ollama can run the layers that do not fit from system memory, and we allow up to 60% of it, but that part reads at about 60 GB/s on dual-channel DDR5 instead of the card's speed. All three figures are our assumptions, not measurements.
A Mac shares one pool of memory, and macOS lets the GPU use only part of it by default. On macOS 26 and later that is about 12.7 GB of a 16 GB Mac, 26.8 GB of 32 GB, 55.7 GB of 64 GB and 115.4 GB of 128 GB, read from public llama.cpp, Ollama and MLX logs. macOS 15 and earlier allow less at these sizes. You can raise the limit until the next restart; the value is in MiB, and macOS needs memory of its own, so leave headroom.
sudo sysctl iogpu.wired_limit_mb=<MiB>Unified-memory PCs work like a Mac. NVIDIA publishes no GPU-only limit for the DGX Spark, so we assume 84%, about 107.5 GB of its 128 GB. What GPU do I have? finds your card's video memory.
What the speed figure means
Pick your machine in the calculator and it also estimates reply speed. Writing each token reads the model's active weights once, so speed follows memory bandwidth. Our formula was fitted to 53 published llama.cpp and MLX runs, with a median error of 6% and every run within 20% (measured on the runs it was fitted to). Speeds are for short chats; long contexts run slower. Where someone measured the same model, file and chip, the measured number replaces the estimate and is labelled measured.
We give no estimate for weights under 2 bits each or for mixture-of-experts models above 120B total parameters, where the formula runs fast: those show a measured speed or none.
What to do with the number
Size for the longest context you will actually use, not the default: Llama 3.1 8B needs 1.1 GB of context memory at 8,192 tokens and 17.2 GB at 131,072. If the total lands above 90% of what your GPU can use, pick the next smaller quant, an 8‑bit cache or a shorter context before buying hardware.
LM Studio
Download a GGUF file from the table and run it with a chat window, on a Mac or a PC.
Free download, closed source. Its terms allow some features to cost extra. Made by another developer, not PocketWebTools.
Get LM Studiollama.cpp
The engine behind most of these numbers. Set the context with -c and the cache type with -ctk and -ctv.
Free and open source (MIT licence). Made by another developer, not PocketWebTools.
Get llama.cppOllama
Runs the same GGUF files from the command line; each model's tag page lists the quant it downloads.
Free and open source (MIT licence). Made by another developer, not PocketWebTools.
Get OllamaPocketWebTools for Mac
Runs 6 open chat models, up to Qwen3.6 35B‑A3B (a 22.4 GB download), plus transcription with speaker labels, text to speech and voice cloning, OCR, photo upscaling and cutouts, all offline on Apple Silicon. It does not generate images, read images in chat, load your own model files, run MLX, link several Macs or serve a local API for coding tools.
It runs Qwen3.6 35B‑A3B on a 32 GB Mac mini M4, measured at 25 to 27 tokens per second with a 32,768-token context window, and checks the fit before every load.
Version 1.0.0. Needs Apple Silicon (M1 or later), macOS 14 Sonoma or later.
Already have a key? Download the app. It asks for the key the first time it opens.
14-day money-back guarantee. Your license key is shown right after checkout and stays on your purchase page; one key works on up to 3 Macs.
How we estimate
Show the formulas, constants and sources
Memory a model needs
Memory = weights + context memory + runtime. Weights are the real download size of the file you pick, read from Hugging Face. When a size is unknown we use parameters x bits per weight / 8, with typical averages per quant type (Q2_K 2.96, Q3_K_M 3.91, Q4_K_M 4.8, UD-Q4_K_XL 4.9, MXFP4 4.25, NVFP4 4.5, Q5_K_M 5.7, Q6_K 6.56, Q8_0 8.5, BF16 16). Runtime is 0.7 GB, or 2% of the weights when that is more, for compute buffers and allocator slack: measured buffers stay near that size whatever the model. Against our own Mac app's measured memory for six models, and Apple's published totals for two models at 65,536 tokens, this lands within 15%. Where someone measured the memory of the exact file, we use that run instead and say so. Checked 2026-10-19.
Context memory (KV cache)
Each token of context stores keys and values: 2 x layers x KV heads x head size x 2 bytes (16‑bit cache, the default in llama.cpp and MLX), read from each model's config.json. Sliding-window layers keep at most their window (2,048 tokens on Muse Spark Open, for example), so they add a fixed amount instead of growing. MLX is confirmed to cap them on Muse Spark Open; whether llama.cpp does for that model is not verified, so its GGUF figures may run low at long context. Models with compressed attention (MLA) store one small latent per layer. Linear-attention layers keep a small fixed state, counted in the runtime share. In llama.cpp you can store the cache at lower precision with --cache-type-k and --cache-type-v (-ctk and -ctv): per element that is q8_0 1.0625 bytes, q4_0 0.5625 bytes, from the size of each type's data blocks. A lower-precision V cache needs flash attention, which llama.cpp turns on by itself unless you switch it off. Measured runs are taken to use the 16‑bit default. Checked 2026-10-19.
- huggingface.co/unsloth/Muse-Spark-Open-816B-A42B/raw/main/config.json
- raw.githubusercontent.com/ml-explore/mlx-lm/v0.32.2/mlx_lm/models/muse_spark.py
- huggingface.co/meta-models/Muse-Glimmer-30B/raw/main/config.json
- github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp
- raw.githubusercontent.com/ggml-org/llama.cpp/master/gguf-py/gguf/constants.py
How much memory a Mac's GPU can use
macOS only lets the GPU use part of a Mac's memory. Apple publishes no formula, so we read the default from public llama.cpp, Ollama and MLX logs where one exists. On macOS 26 and later: 8 GB: 2/3, about 5.7 GB (medium evidence); 16 GB: 0.74, about 12.7 GB (strong evidence); 18 GB: 0.74, about 14.3 GB (medium evidence); 24 GB: 0.74, about 19.1 GB (strong evidence); 32 GB: 0.78, about 26.8 GB (strong evidence); 36 GB: 0.78, about 30.2 GB (medium evidence); 48 GB: 0.78, about 40.2 GB (strong evidence); 64 GB: 0.81, about 55.7 GB (strong evidence); 96 GB: 0.81, about 83.5 GB (weak evidence); 128 GB: 0.84, about 115.4 GB (strong evidence); 192 GB: 0.84, about 173.2 GB (weak evidence); 256 GB: 0.87, about 239.1 GB (weak evidence); 512 GB: 0.87 (assumed), about 478.3 GB (unconfirmed evidence). Why the weaker rows are weaker: 96 GB: One log, from 2026-09-30. 192 GB: One M2 Ultra log, from 2025-10-30. 256 GB: Two M3 Ultra logs, possibly from the same person. 512 GB: No confirmed default. Logs show 464 GiB (about 0.906), but the same value appears on a macOS 15 Mac, so it is probably a manual override. We use 0.87, the 256 GB ratio, which may understate a 512 GB Mac by about 20 GB. On macOS 15 or earlier it was 2/3 up to 32 GB and 3/4 above. The limit can be raised with sudo sysctl iogpu.wired_limit_mb=<MiB>; it resets on restart and macOS needs memory of its own, so leave headroom. The setting cannot go past a second limit, the kernel's cap on all locked memory, which Apple's open-source macOS kernel sets by memory size (16 GB: 13.1 GB; 32 GB: 27.1 GB; 64 GB: 56.3 GB; 128 GB: 116.8 GB; 256 GB: 241.9 GB; 512 GB: 515.4 GB) and two measured Macs match exactly. That cap sits a little above the default at every size, so the default is what a model gets, and we suggest the setting only when a model fits under the cap. It may still fail. Checked 2026-10-19.
- github.com/geerlingguy/ai-benchmarks/issues/55
- github.com/ollama/ollama/issues/13460
- github.com/ollama/ollama/pull/14604
- github.com/waired-ai/waired-agent/issues/857
- github.com/ggml-org/llama.cpp/issues/20704
- github.com/AshiqIqbal1/Whisperlet/pull/81
- github.com/OpenCSGs/csglite/issues/73
- github.com/ggml-org/llama.cpp/pull/21048
- github.com/ollama/ollama/issues/15545
- github.com/ggml-org/llama.cpp/pull/16669
- github.com/vllm-project/vllm-metal/pull/764
- github.com/ggml-org/llama.cpp/pull/16634
- github.com/r0shi/harborclerk/issues/684
- github.com/ggml-org/whisper.cpp/pull/3822
- github.com/red-hat-ai-dev/vLLM-zero-to-hero-pt1/issues/7
- github.com/ggml-org/llama.cpp/issues/20141
- github.com/ggml-org/llama.cpp/issues/19679
- github.com/ggml-org/llama.cpp/issues/20011
- github.com/ggml-org/llama.cpp/issues/21013
- github.com/mdegans/drama_llama/issues/125
- github.com/ggml-org/llama.cpp/issues/16475
- github.com/ggml-org/llama.cpp/issues/17607
- github.com/evanwtf/local-llm/issues/30
- github.com/ollama/ollama/issues/12847
- github.com/ollama/ollama/issues/12900
- github.com/ggml-org/llama.cpp/issues/19335
- github.com/ggml-org/llama.cpp/issues/17291
- github.com/ggml-org/llama.cpp/issues/16093
- github.com/ggml-org/llama.cpp/discussions/2182
- github.com/ml-explore/mlx-lm
- github.com/waired-ai/waired-agent/issues/857
- github.com/apple-oss-distributions/xnu/blob/main/osfmk/vm/vm_kern.c
- github.com/apple-oss-distributions/xnu/blob/main/osfmk/vm/vm_page.h
- github.com/osaurus-ai/vmlx-swift/pull/608
Several Macs linked together
For a group of Macs with the same memory, we add up what each one's GPU can use: two 256 GB Macs on macOS 26 and later come to about 478.3 GB. Assumes the model splits evenly across Macs of the same memory size (as exo and MLX distributed do) and that linking them costs no memory on any one Mac. Neither is measured. We give no estimated speed for a group; only published runs, each with its number of Macs. Checked 2026-10-19.
How much memory a PC can use
On a graphics card we take its video memory minus 0.5 GB for the display. Runtimes like llama.cpp, LM Studio and Ollama can put layers that do not fit into system memory; we allow up to 60% of it, and any model that needs it is graded Tight. Both numbers are our assumptions, not measurements. AMD documents up to 96 GB of GPU memory on a 128 GB Ryzen AI Max machine; on smaller ones we assume the same 75% share. NVIDIA publishes no GPU limit for DGX Spark, so we assume 84% of its memory, the share macOS gives a 128 GB Mac. Checked 2026-10-19.
Reply speed
Writing each token reads the model's active weights once, so speed follows memory bandwidth: seconds per token = active GB / (bandwidth x 0.85) + a fixed cost per token of 2 ms on Apple for standard models, 5 ms for hybrid linear-attention models (Qwen3.5 and 3.8), 6 ms for mixture of experts; 1.3 ms on NVIDIA and DGX Spark for standard models and 2 ms for mixture of experts. Active GB comes from the file's tensor table where we computed it, else from a similar file of the same model whose table we read, scaled by file size, else from the active parameters at the file's own bits per weight. When a PC spills into system memory, that part reads at 60 GB/s x 0.6 (assumed dual-channel DDR5). Fitted to 53 published llama.cpp and MLX runs, it had a median error of 6% and every point within 20%; that fit is in-sample. AMD and Intel cards borrow the NVIDIA values and are marked rough. Speeds are at short context; long chats run slower. Checked 2026-10-19.
- github.com/ggml-org/llama.cpp/discussions/4167
- github.com/ggml-org/llama.cpp/discussions/15396
- github.com/ggml-org/llama.cpp/discussions/15013
- github.com/ggml-org/llama.cpp/blob/master/benches/dgx-spark/dgx-spark.md
- github.com/ggml-org/llama.cpp/pull/27390
- machinelearning.apple.com/research/mlx-distributed-inference
- github.com/john-rocky/apple-silicon-llm-bench
Why speeds are a range
The top of the range is the formula above. The bottom is 65% of it on M1 and M2 chips, where our own Mac app ran about a third under the formula on two mid-size models, and 85% everywhere else. The M1 and M2 rule rests on those two points only. Checked 2026-10-19.
When we do not estimate speed
We show no estimated speed for weights under 2 bits each, which are limited by computation rather than bandwidth (a 1‑bit 27B model ran at about a quarter of the bandwidth limit), or for mixture-of-experts models above 120B total parameters, where the formula ran 40% fast on Muse Spark Open against a measured 18 tokens per second. Those rows show measured speeds where someone published them, and "not measured" otherwise, and a model with no speed figure is graded Tight at best. Checked 2026-10-19.
Measured beats estimated
When someone published a speed for the same model, quant and chip, we show that number, labelled measured, with its source and caveats, instead of the formula. A run on another runtime (Meta's ExecuTorch numbers for a GGUF, for example) or one that does not say which GPU-core version of a chip it used is shown for comparison only. Our own measurements come from PocketWebTools for Mac on a 16 GB M1 Pro and a 32 GB Mac mini M4. A speed measured at a context more than 4 times shorter than the one you picked (or at an unstated one, which usually means a short test) is labelled measured at short context, since long chats run slower. Measurements of several Macs linked together never stand in for one Mac, and each is shown with its number of Macs. Model data checked 2026-10-19.
Grades
Won't fit: the model needs more than the GPU can use (plus the allowed system memory on a PC). Tight: it fits but needs more than 90% of that memory, spills into system memory, replies under 8 tokens per second, or has no speed figure for that machine. Runs well: everything else. Every grade states the quant, the context length (8,192 tokens unless you change it) and the memory figure it assumed. Speech, voice and image models are graded on memory only. Checked 2026-10-19.
Time to the first word
Reading your prompt is limited by computation, not bandwidth. We scale each chip's published llama.cpp prompt speed on a 7B model by 6.6B / the model's active parameters, x 0.75 for mixture of experts. It matched measured runs within 8% for standard models. Checked 2026-10-19.
Questions
- Is 24 GB of VRAM enough for a local LLM?
- Yes for models up to about 30B at 4‑bit, though it gets tight at long context: Qwen3 30B‑A3B at Q4_K_M needs 22.5 GB at 32,768 tokens of the 23.5 GB a 24 GB card keeps free, which we grade Tight. Llama 3.3 70B at Q4_K_M needs 46.1 GB, so about half runs from system memory and replies drop to an estimated 1.4 to 1.7 tokens per second.
- Is 32 GB of VRAM overkill?
- Not for long context or 8‑bit files: Gemma 3 27B needs 28.4 GB at 131,072 tokens, close to the 31.5 GB a 32 GB card keeps free, and Qwen3 32B at Q8_0 needs 37.7 GB at 8,192, more than it holds. For 8B to 14B models at 4‑bit, a 16 GB card is enough (Llama 3.1 8B 6.7 GB, Qwen3 14B 11.0 GB at 8,192 tokens).
- How much VRAM do I need for a 32B model?
- About 22.6 GB at Q4_K_M with 8,192 tokens of context (Qwen3 32B), 29.0 GB at 32,768 tokens, and 37.7 GB at Q8_0 with 8,192 tokens.
- How do I calculate VRAM for LLM inference?
- Add the quantized file size, the context memory (2 x layers x KV heads x head size x 2 bytes x tokens for standard attention) and about 0.7 GB for the runtime. The calculator above does it from each model's real file and config, or from the numbers you type in.
- Does this work for Ollama models?
- Yes. Ollama runs GGUF files, so pick the same model and the quant your Ollama tag downloads (its tag page on ollama.com lists it), and the context length you set.
- Why does llama.cpp use less context than the model supports?
- llama.cpp's server starts from the model's own longest context, but its fit option, on by default, lowers settings you did not set until the model fits in memory, down to 4,096 tokens. Set -c yourself to get the context you sized for here. Both settings are in the llama.cpp server options.
Where to go next
First version: 48 language models with their real file sizes and attention layouts, llama.cpp's 8‑bit and 4‑bit cache types, and what the GPU can use at every Mac memory size.