Transformers

Fundamentals

• Transformer
• Embedding
• Tokenization
• Positional Embeddings

Attention Stack

• Self-Attention
• Multi-Head Attention
• Attention Mask
• Bidirectional Attention
• Cross-Attention

Transformer Block

• How Transformer Block Works
• Feed-Forward Layer
• Layer Normalization
• Classification Head

Architectures

• Encoder-only vs Decoder-only vs Encoder-Decoder
• Decoder
• Encoder-Decoder Architecture
• Autoregressive Language Model

Production And Inference

• Transformer Inference Optimization
• Model Benchmarking
• Knowledge Distillation
• Quantization
• Pruning
• ONNX
• ONNX Runtime

Sources

• NLP Transformers - Chapter 03 - Transformer Anatomy
• NLP Transformers - Chapter 08 - Making Transformers Efficient in Production
• Hands-On LLM - Chapter 03 - Looking Inside Large Language Models

Questions

• When to Prefer Pre-LN Over Post-LN
• How Do Positional Embeddings Scale Context Length