Inference
VRAM & speed
Whether a model fits, and how fast it will decode once it does. Uses the real attention shape of each model — grouped-query head counts, sliding windows, MoE routing, MLA latents — because those change the answer by multiples, not percentages.
—
Total memory
—
Decode
—tok/s
Prefill
—tok/s
Max context
—
that fits entirely in memory
Memory breakdown
0
Table view & working
| Component | Size | Share | How it is derived |
|---|
Quantisation ladder
| Quant | Weights | Total | Fits | Decode | Max ctx |
|---|
Rows are the same model at the same context on the same hardware — only the weight precision changes. The highlighted row is your current selection.
Context scaling
| Context | KV cache | Total | Fits | Decode | Time to first token |
|---|
What this assumes
- Decode is memory-bound. Speed is bytes read per token divided by achieved bandwidth, where bytes read is the active weights plus the whole KV cache. That last term is why a model runs slower at 100k context than at zero, which most calculators never show you.
- Prefill is compute-bound. Two FLOPs per active parameter per token, against dense FP16 tensor throughput at a realistic 45% MFU. Marketing FLOPS numbers assume 2:1 sparsity that no LLM runtime achieves; the figures here do not.
- Bits-per-weight are measured, not nominal. Q4_K_M is 4.83 bpw in real files, not 4.00 — K-quants carry block scales and keep some tensors at higher precision. Using the nominal width underestimates every K-quant by 15–20%.
- Sliding-window and MLA models are modelled properly. Gemma 3 keeps a 1024-token window on five of every six layers, and DeepSeek compresses KV to a 576-value latent. Treating either as plain GQA overstates the cache by up to 10×.
- Expect ±20%. These are calibrated first-principles estimates, not benchmarks. Runtime, driver, kernel and flash-attention choices all move the number. Use it to decide what to download; use a real benchmark to decide what to ship.