tools.ml.box

Serving

Batch throughput

How many people one GPU can serve at once before the experience gets bad. Weights are read once per decode step and the KV cache is read per sequence, so batching multiplies total throughput while barely touching per-user speed — until the cache runs out of room.

Workload

Hardware

—

Concurrent users

—

Total throughput

—tok/s

Per user

—tok/s

Cost / M tokens

—

Where the memory goes

0

    Batch ladder

    Throughput and latency at each concurrency level
    Concurrent Per user Total KV used Bound by $ / M tok

    What this assumes

    • Weights once, KV per sequence. A decode step reads the active weights a single time no matter how many sequences are in flight, then reads each sequence's cache. That asymmetry is the whole reason batching works, and why long contexts destroy it — at 32k the cache dwarfs the weights and the per-step read scales with users again.
    • Concurrency is usually memory-limited, not compute-limited. On most cards you run out of room for KV caches long before you run out of FLOPs. The ladder marks which constraint binds at each level.
    • 10% of VRAM is held back. vLLM defaults gpu_memory_utilization to 0.90, and a paged allocator needs slack to avoid preemption thrash under bursty arrival. Sizing to 100% of free memory builds a server that OOMs in its first bad minute.
    • This is steady-state throughput, not p95 latency. It assumes every slot is full and arrival is smooth. Real traffic is bursty, queueing adds delay before a request starts decoding, and prefill competes with decode for the same GPU. Size for less than this.
    • Decode MFU is 25%. Batched decode GEMMs are skinny — batch × hidden against hidden × hidden — so they never reach the arithmetic intensity a long prompt does, even at large batch.