Overview Text & Messages Chat Template Tokenization Normalization Pre-Tokenization & BPE WordPiece / Unigram Token IDs Embedding Text Embedding & Retrieval Transformer Block Attention Q / K / V Scaled Dot-Product Attention MHA / MQA / GQA Position / RoPE FFN SwiGLU / MoE Norm Residual Stream LM Head Logits Training Loss Generation KV Cache Sampling / Argmax Next Token Autoregressive Generation Token IDs → Embedding Pre-trained Language Models PLM Encoder-only BERT RoBERTa ELECTRA Modern Encoders Encoder-Decoder T5 / FLAN-T5 BART NLLB / Whisper / T5Gemma Decoder-only GPT-1 GPT-2 GPT-3 / ICL LLaMA / Mistral / Mixtral Contents 01 Lecture 2 02 Lecture 3 03 Lecture 4