modern arch

  1. 00The mapFrom the 2017 block to a 2024 LLM block
  2. 01RoPE, properlyFrequencies, caches, conventions and long context
  3. 02RMSNormRescale, skip the re-centering
  4. 03SwiGLUA gated MLP with a smooth switch
  5. 04Grouped-query attentionMany queries, few keys and values
  6. 05SDPAOne call, three backends
  7. 06FlashAttentionExact attention that respects memory
  8. 07The modern decoder blockEvery upgrade, assembled

chapter 03

SwiGLU

The Day 1 MLP widened the vector, bent it with GELU, and narrowed it back.

Two MLPs

vanilla: Wdown · GELU(Wup x)     SwiGLU: Wdown · ( SiLU(Wgate x) ⊙ Wup x )

Vanilla MLP vs SwiGLU MLP

ReLU, GELU, SiLU

SiLU(x) = x · σ(x) = x / (1 + e−x)

The gate in action

Gating, element by element

The gate surface

Sizing the hidden layer

From 4d to 8d/3, then rounded

h = multiple_of · ⌈ ffn_mult · ⌊8d/3⌋ / multiple_of ⌉

The GLU family

What the paper found

Fusing gate and up into one matmul

Exceptions and sharp edges

Hidden size is not 4d

read intermediate_size from the config.

Swapped halves

check the loading code's split order.

More activation memory

activation checkpointing, or fused kernels that recompute.

No biases

bias=False on all three projections.

Names vary

map names explicitly when converting.