modern arch

  1. 00The mapFrom the 2017 block to a 2024 LLM block
  2. 01RoPE, properlyFrequencies, caches, conventions and long context
  3. 02RMSNormRescale, skip the re-centering
  4. 03SwiGLUA gated MLP with a smooth switch
  5. 04Grouped-query attentionMany queries, few keys and values
  6. 05SDPAOne call, three backends
  7. 06FlashAttentionExact attention that respects memory
  8. 07The modern decoder blockEvery upgrade, assembled

octlm · day 2 · modern decoder architecture

Modern decoder architecture

Day 1 built the textbook block: learned positions, LayerNorm, a GELU MLP, multi-head attention written by hand.

From GPT-2 to Llama, part by part

When each idea arrived

What each upgrade buys

A real modern config, annotated

Chapters

chapter 01 · act 1RoPE, properlyFrequencies, caches, conventions and long context
chapter 02 · act 1RMSNormRescale, skip the re-centering
chapter 03 · act 1SwiGLUA gated MLP with a smooth switch
chapter 04 · act 2Grouped-query attentionMany queries, few keys and values
chapter 05 · act 2SDPAOne call, three backends
chapter 06 · act 2FlashAttentionExact attention that respects memory
chapter 07 · act 3The modern decoder blockEvery upgrade, assembled