Transformer prefill & decode

Two layers, one attention head. Follow the hidden states and each layer’s KV cache.

Prompt: A B C · T = 3 · head width = model width D · batch dimension omitted

Explore the inference steps

Select a step to see how the diagram and KV caches change.

Computed / written this step Reused from earlier steps