The memory arithmetic

Backing derivations for the series' ground rules. Six numbers come out of this file: 28 KiB of cache per token, 3.09 GB of weights, a ridge point of 208 FLOP per byte, a batch 1 ceiling of 194 tok/s, roughly 19 GiB free on the card, and a 2 GiB pool cap. The formulas are from kipply's Transformer Inference Arithmetic (2022).

Weights

Parameters times 2 bytes, at 16 bit precision:

1.54e9 x 2 = 3.09 GB   (2.88 GiB)

That matches the weights file on disk to within the size of its header, which is a useful check that the parameter count is right rather than rounded off a model card.

KV cache per token

Every token that has already been generated leaves behind one key vector and one value vector in each layer, and every later token reads them. The cost per token is

2 x 2 x n_layers x n_kv_heads x head_dim

where the first 2 is the key and value pair and the second is bytes per 16 bit value. Qwen2.5 uses grouped query attention, so the head term is the 2 key value heads, not the 12 attention heads:

2 x 2 x 28 x 2 x 128 = 28,672 bytes = 28 KiB per token

Grouped query attention is doing the heavy lifting here. Under full multi head attention the same model would spend 2 x 2 x 28 x 12 x 128 = 168 KiB per token, 6x more. Cheap keys and values mean capacity is not the first thing that binds on this model, which changes which optimizations are worth building and in what order.

The ridge point

Arithmetic intensity is operations performed per byte moved. A chip's ridge point is the intensity at which its arithmetic units and its memory system would finish at the same moment, so it is peak compute over peak bandwidth:

125 TFLOPS / 600 GB/s = 208 FLOP per byte

Work below 208 FLOP per byte leaves the arithmetic units idle waiting on data. Work above it leaves the memory system idle instead. A forward pass over n tokens has an intensity of about n, because the weights are read once per pass however many tokens ride through it, so the line falls at 208 tokens. Prefill over a long prompt sits above it and every decode step sits far below.

The ceiling on one sequence

A decode step produces one token and has to read every weight to do it. Reading them once is therefore the fastest that token can possibly appear:

3.09 GB / 600 GB/s = 5.15 ms per token = 194 tok/s

This is a ceiling for a single sequence, not for the card. Running B sequences through one pass reads the same weights once and produces B tokens, which is why batching raises the aggregate figure and this number does not move.

What the card actually has

Measured on the A10 rather than read off the spec sheet:

The 2.88 GiB weights figure matches the arithmetic above exactly. Roughly 0.3 GiB of the card is not visible to CUDA at all, and the allocator holds about 0.2 GiB beyond the weights themselves.

Why the pool gets capped

19 GiB at 28 KiB per token is about 710,000 token positions, or around 340 concurrent 2048 token sequences. On a 1.5B model, memory would simply never bind, and any memory management work would have nothing to show for itself. A larger model would be sharded across several cards and would not have this much room to spare, so a cap is the way to get a realistic ratio of cache size to model size on one small GPU.

Capping the pool at 2 GiB admits 74,898 token positions:

vLLM exposes the same knob, as a fraction of total GPU memory. Sizing the pool deliberately and reporting what happens at the ceiling is more informative than running with 19 GiB free and never reaching one.