Deriving MFU and MBU from first principles

A complete derivation of the FLOP counts, byte counts and utilization figures reported in every part. Everything below follows from the model architecture, the hardware specification, and one measured quantity: wall clock time.

Appendix to Part 1.

---

I. Givens

G1. Model architecture (Qwen2.5-1.5B)

Note H · d_h = 12 × 128 = 1,536 = d, and H_kv · d_h = 2 × 128 = 256.

G2. Hardware (NVIDIA A10)

G3. Workload

G4. Attention backend

attn_implementation is pinned to sdpa in infer/runtime.py, which is the value transformers selected for this baseline before it was pinned.

PyTorch's SDPA is a dispatcher, not an implementation: at each call it selects among flash, mem efficient, and math backends. The first two skip causally masked score entries; math computes the full matrix and masks afterwards. Flash and mem efficient are eligible only when no explicit mask tensor is passed and is_causal=True (transformers/integrations/sdpa_attention.py).

Observed arguments at every one of the 28 layers, captured by intercepting scaled_dot_product_attention:

attn_mask   = None
is_causal   = True
enable_gqa  = True
q_length   == kv_length

This excludes the math backend, so the masked half is skipped. Flash versus mem efficient is not distinguished, and does not need to be: both skip it, so the FLOP count is identical either way.

These arguments are chosen by transformers in Python, independent of the device, so this holds for the recorded A10 runs.

---

II. Definitions

D1. FLOP. One floating point operation: a single multiply or a single add.

D2. Token forward. One token passed through the model one time. The natural unit of work for an engine whose cost scales with tokens processed rather than tokens emitted.

D3. Achieved throughput. Total FLOPs performed during a run divided by that run's wall clock duration.

        F(P)
R  =  ────────
         t

D4. MFU (model FLOPs utilization). Achieved throughput as a fraction of hardware peak.

MFU  =  R / Π

D5. Sequence length at step i. In an engine with no KV cache, step i processes the whole sequence built so far:

n_i  =  P + i        for  i = 0, 1, …, N−1

---

III. Lemmas

L1. Cost of a matrix multiply

Let C = A @ B with A : [n × d] and B : [d × m].

C has n · m entries. Each is a dot product of two length-d vectors, requiring d multiplies and d − 1 adds, i.e. 2d − 1 ≈ 2d FLOPs.

FLOPs(matmul)  =  2 · n · d · m

L2. Cost of a linear layer

A linear layer holds a weight matrix [d_in × d_out], therefore d_in · d_out parameters. Applying it to n tokens is the matmul [n × d_in] @ [d_in × d_out]. By L1:

FLOPs(linear)  =  2 · n · d_in · d_out  =  2 · n · (parameters in the layer)

Corollary. Summed over every linear layer, pushing n tokens through a network of N_params parameters costs 2 · n · N_params.

L3. Parameter census

Counted directly from G1, one transformer layer contains:

Note the effect of GQA: k_proj and v_proj map into H_kv · d_h = 256 dimensions rather than d = 1536, making them 6× smaller than under multi head attention.

Totalling across the model:

body       =  46,797,824 × 28  +  1,536   =  1,310,340,608
                                (final norm)

embedding  =  V × d  =  151,936 × 1,536   =    233,373,696

total      =                                 1,543,714,304

Verification against disk. At 2 bytes per fp16 parameter the tensor payload is 1,543,714,304 × 2 = 3,087,428,608 bytes. The file model.safetensors measures 3,087,467,144 bytes; the difference of 38,536 bytes is the safetensors JSON header, which precedes the tensor data.

Define for use below:

B  =  1,310,340,608     (body parameters)
E  =    233,373,696     (embedding = LM head)

L4. Cost of the body, per forward pass

By the corollary to L2, restricted to body parameters:

FLOPs_body(n)  =  2 · B · n

L5. Cost of the LM head, per forward pass

The engine passes logits_to_keep = 1, so the vocabulary projection is applied to the final position only, not to all n positions:

FLOPs_head  =  2 · 1 · d · V  =  2 · E  =  466,747,392

This term is independent of n. Applied to every position instead it would be 2 · E · n, which at n = 4096 is 1.9 TFLOP per pass rather than 0.0005 TFLOP.

L6. Cost of attention, per forward pass

Attention contains two matmuls with no associated parameters, so L2 does not apply. Per layer, per query head:

scores = Q @ Kᵀ  :  [n × d_h] @ [d_h × n]   →   2 · n · d_h · n  =  2 d_h n²
out    = A @ V   :  [n × n]   @ [n × d_h]   →   2 · n · n · d_h  =  2 d_h n²

Over H query heads and L layers:

FLOPs_attn(n)  =  4 · d_h · H · L · n²  =  4 · d · L · n²  =  172,032 · n²

GQA does not reduce this term. There remain H = 12 query heads producing 12 distinct score matrices; the H_kv = 2 key/value heads are broadcast across them. GQA reduces storage and memory traffic, not attention arithmetic. Under enable_gqa=True (G4) the broadcast happens inside the kernel rather than by materializing an expanded tensor, which saves memory traffic but no FLOPs.

Causal correction. Position i attends only to positions j ≤ i, so only (n² + n)/2 of the n² score entries are required. The backend in use (G4) skips the masked half rather than computing and discarding it, so the term is halved:

FLOPs_attn(n)  =  ½ · 4 · d · L · n²  =  86,016 · n²

The residual approximation is tile granularity: blocks straddling the diagonal are computed in full. That error is O(n) against an O(n²) term and is negligible at these lengths.

Define:

A  =  ½ · 4 · d · L  =  86,016

L7. Summation across the generation loop

By D5, over N = 256 steps:

Σn   =  Σ (P + i)   =  N·P  +  Σi   =  256P + 32,640

Σn²  =  Σ (P + i)²  =  N·P² + 2P·Σi + Σi²
                    =  256P² + 65,280P + 5,559,680

using the closed forms

Σi   =  (N−1)N/2        =  255 · 256 / 2        =    32,640
Σi²  =  (N−1)N(2N−1)/6  =  255 · 256 · 511 / 6  = 5,559,680

---

IV. Theorem

Total FLOPs for one run at prompt length P, generating N = 256 tokens without a KV cache:

F(P)  =  2·B·Σn  +  2·E·N  +  A·Σn²
         ───────     ──────     ──────
          body        head       attention

with B = 1,310,340,608, E = 233,373,696, A = 86,016, and Σn, Σn² as given by L7.

By D3 and D4, for a measured wall clock time t:

MFU  =  F(P) / (t · Π)

---

V. Computation

At P = 4096

Sums (L7):

Σn   =  256 × 4096 + 32,640
     =  1,048,576 + 32,640
     =  1,081,216

Σn²  =  256 × 4096²  +  65,280 × 4096  +  5,559,680
     =  4,294,967,296  +  267,386,880  +  5,559,680
     =  4,567,913,856

Terms (Theorem):

body  =  2 × 1,310,340,608 × 1,081,216  =  2.8335e15  =  2833.5 TF
head  =  2 ×   233,373,696 × 256        =  1.1949e11  =     0.1 TF
attn  =         86,016 × 4,567,913,856  =  3.9290e14  =   392.9 TF
                                                        ──────────
                                             F(4096)  =  3226.5 TF

Throughput (D3), with t = 71.15 s (mean of 5 recorded runs):

R  =  3.2265e15 / 71.15  =  4.535e13  =  45.4 TFLOPS

Utilization (D4):

MFU  =  45.4 / 125  =  36.3%

At P = 16

Σn   =  256 × 16 + 32,640                              =     36,736
Σn²  =  256 × 256 + 65,280 × 16 + 5,559,680            =  6,669,696

body =  2 × 1,310,340,608 × 36,736      =   96.3 TF
head =                                       0.1 TF
attn =   86,016 × 6,669,696             =    0.6 TF
                                          ─────────
                               F(16)    =   97.0 TF

R    =  9.70e13 / 5.61  =  1.73e13      =  17.3 TFLOPS
MFU  =  17.3 / 125                      =  13.8%

---

VI. Results

---

VII. Bandwidth

MFU answers whether the arithmetic units are busy. For decode they are not, and the question that carries information is whether the memory bus is.

D6. MBU (model bandwidth utilization). Bytes that must cross the memory bus during a run, divided by what the bus could have delivered in the same time:

MBU  =  (B / t) / β

L8. Bytes per forward pass

Every parameter is read once per forward pass, however many tokens ride through it. At 2 bytes each:

W  =  2 × 1,543,714,304  =  3,087,428,608 B  =  3.09 GB

An engine with no cache holds no other persistent state, so a naive pass moves W and nothing else. An engine with a cache also reads the whole cache back every decode step. At step j the cache holds P + j positions at 28,672 bytes each (G1, and the same figure derived in the arithmetic appendix), so averaged across N steps:

B_cached  =  W  +  (P + N/2) · 28,672

This counts weights and cache only. Activations are ignored: they are small at batch 1, and most of their traffic never leaves on chip memory. Nothing here observes actual DRAM traffic, so every byte figure is derived in the same sense the FLOP figures are.

Results

Per decode step, with step time taken as decode time / (N − 1):

The naive engine's bytes per step are constant while its step time grows, so its MBU falls away. The cached engine's step time is constant, so its MBU sits between 22.7% and 24.5% at every prompt length. Neither engine gets close to 600 GB/s, and the reason is in Part 2: the bus is idle most of the step, waiting for the host.