- Explain the Transformer architecture end to end — embedding, positional encoding, multi-head self-attention, feed-forward blocks, residuals, and the output head — as introduced in Attention Is All You Need (2017).
- Contrast Transformers with classic deep learning (RNNs, LSTMs, CNNs, MLPs): why attention replaced recurrence, and what Transformers kept from deep learning.
- Trace the forward pass of a decoder-only LLM like GPT, including masked self-attention and autoregressive next-token generation.
- Understand recent innovations — RoPE, GQA/MQA, FlashAttention, Mixture-of-Experts, long-context tricks, and quantization — and which production models use them.
- Run hands-on labs: a softmax/sampling sandbox, an attention-matrix explorer, and a model-size calculator.
- Compare today's assistants — ChatGPT, Claude, Grok, and Meta AI — as different products built on the same Transformer foundation.
| # | Session | Focus | Time |
|---|---|---|---|
| 1 | From Deep Learning to Attention | RNN/CNN limits, why self-attention, the 2017 leap | 2 h |
| 2 | The Transformer Block | Embedding, positional encoding, residual + LayerNorm | 2 h |
| 3 | Self-Attention In Depth | Q/K/V, scaled dot-product, masking, multi-head | 2 h |
| 4 | Decoder-Only LLMs & Generation | Causal masking, softmax, sampling, autoregression | 2 h |
| 5 | Recent Innovations | RoPE, GQA/MQA, FlashAttention, MoE, long context | 2 h |
| 6 | Efficiency & Deployment | KV-cache, quantization, distillation, serving | 2 h |
| 7 | Frontier Models & Capstone | ChatGPT/Claude/Grok/Meta AI, project work | 2 h |
- Python 3.9+ with
pip install torch transformers numpy matplotlib. - A notebook environment — Jupyter, Google Colab, or Kaggle (free GPU helps but is not required for the labs here).
- A Hugging Face account to pull pretrained models such as
gpt2/distilgpt2. - Optional —
bertvizfor attention visualization, and any LLM API key for the capstone.
- Attention Is All You Need — Vaswani et al., 2017 (the founding paper).
- "Transformers Architecture and Recent Innovations" — R. Raghav, LinkedIn (course framing).
- The Illustrated Transformer — Jay Alammar · Transformer Explainer — Georgia Tech (Polo Club).
- Hugging Face Transformers documentation & course.
Foundation Keywords
Everything you need before reading the architecture — what a token and an embedding are, how attention lets words "look at" each other, and how a stack of these blocks becomes a language model. No prior NLP experience assumed.
Key idea: a Transformer is the same small block repeated many times. Each repeat lets every token gather a little more context, until the model "understands" the sentence well enough to predict what comes next.
Deep Learning vs Transformers
Transformers are deep learning — but a specific kind that abandoned recurrence and convolution in favour of attention. This page contrasts the classic architectures (RNN, LSTM, CNN, MLP) with the Transformer, and shows what stayed the same.
| Dimension | Classic Deep Learning (RNN / LSTM / CNN) | Transformer |
|---|---|---|
| Sequence processing | Sequential — one timestep at a time (RNN/LSTM) | Parallel — the whole sequence at once |
| Long-range dependencies | Hard — signal fades over distance (vanishing gradients) | Direct — any token can attend to any other in one step |
| Training speed | Slow — recurrence blocks GPU parallelism | Fast — fully parallel across positions |
| Core operation | Recurrence (RNN), convolution (CNN) | Self-attention (scaled dot-product) |
| Inductive bias | Strong — locality (CNN), order (RNN) | Weak — must learn structure from data + positional encoding |
| Compute cost | Linear in sequence length | Quadratic O(n²) in sequence length (the main cost) |
| Data appetite | Works with less data | Hungry — shines at large scale |
| Dominant use today | Edge/timeseries, vision (CNN still strong) | Language, multimodal, increasingly vision & audio |
- They are still neural networks trained by backpropagation and gradient descent on a loss.
- Stacked layers + nonlinearities — depth still builds abstraction, just with attention instead of recurrence.
- Residual connections & normalization — borrowed straight from ResNets and earlier deep-learning work to train deep stacks.
- Embeddings & softmax — the input and output machinery predates Transformers.
- The MLP — every block contains a classic feed-forward network doing most of the "thinking".
One-line summary: RNNs and CNNs ask "what's nearby in time or space?" Transformers ask "what's relevant anywhere?" — and answer it for every token, in parallel.
Running example: the prompt "Data visualization empowers users to" → 6 tokens. The hidden size below is GPT-2 small's d_model = 768, with 12 blocks and 12 heads.
Every token produces three vectors: a Query (what am I looking for?), a Key (what do I offer?), and a Value (what will I pass on?). Attention scores each token against every other, turns the scores into weights, and blends the Values accordingly.
Why divide by √d_k? Dot products grow with dimension; without scaling, softmax saturates and gradients vanish. The √d_k keeps scores in a sane range.
Instead of one attention, run h of them in parallel on d_model/h-sized slices. Each head can specialise — one tracks syntax, another long-range reference — and their outputs are concatenated and projected back.
Try the Self-Attention Sandbox in the Labs tab to see per-head weight patterns and the causal mask in action.
Interactive Labs
Three live demonstrations that run entirely in this page — no install. Turn the dials and watch the math respond.
A frozen set of candidate next-tokens with fixed logits. Adjust temperature, top-k, and top-p and watch the probability distribution reshape — exactly what a model does at every generation step.
An attention weight matrix for a short sentence. Each row is a query token; brighter cells = more attention paid to that key token. Toggle the causal mask and switch heads to see different patterns.
Weights here are illustrative (deterministic), generated to show typical head behaviour — not from a live model.
Punch in an architecture and estimate its parameter count — the same arithmetic that separates GPT-2 (124M) from frontier models.
Each snippet has a real, common Transformer/attention bug. Predict the problem, then reveal the diagnosis.
Each builds on the last, from attention-from-scratch to a frontier-model comparison. Submit a notebook plus a short write-up.
- Hallucination: a Transformer predicts plausible tokens, not facts. Verify outputs; ground them with retrieval (RAG) where correctness matters.
- Bias & representation: models inherit biases from training data. Evaluate across groups before deploying in hiring, lending, or moderation.
- Context & privacy: anything in the prompt may be logged or echoed. Don't paste secrets or personal data into shared models.
- Compute & cost: attention is O(n²); long contexts and large models carry real energy and dollar costs — measure before scaling.
- Attribution & licensing: generated text/code can resemble training data. Respect licences and disclose AI assistance where required.
- Evaluation, not vibes: benchmark on YOUR task. Public leaderboards rarely match your domain.
Frontier Assistants Compared
ChatGPT, Claude, Grok, and Meta AI are four products built on the same decoder-only Transformer foundation. What differs is scale, training data, alignment method, tools, and openness. Snapshot as of June 2026 — this field moves fast, so always re-check current versions.
| Aspect | ChatGPT | Claude | Grok | Meta AI |
|---|---|---|---|---|
| Maker | OpenAI | Anthropic | xAI | Meta |
| Flagship (Jun 2026) | GPT-5.5 | Claude Opus 4.8 | Grok 4.3 | Llama 4 |
| Architecture | Decoder-only Transformer (all four) — differences are in scale, data, alignment & tooling | |||
| Signature strength | Broad all-rounder; tools, images, agents | Coding, long-form, careful document work; tops intelligence/coding indexes | Real-time X data; reasoning-first; permissive | Open-weight & self-hostable; baked into Meta apps |
| Alignment approach | RLHF + safety tuning | Constitutional AI + RLHF | RLHF; lightest guardrails | RLHF on open Llama base |
| Openness | Closed weights | Closed weights | Mostly closed (some older weights opened) | Open weights (Llama licence) |
| Context window | Large (long-context tiers) | Large (long-context, Projects) | ~1M tokens | Very large (Llama 4 Scout up to ~10M) |
| Where you use it | chatgpt.com, API, apps | claude.ai, API, Code/Cowork | X (Twitter), grok.com, API | WhatsApp, Instagram, FB, Ray-Ban, API |
Model names/versions change frequently; figures reflect mid-2026 reporting and should be re-verified for current work.
The teaching point: none of these "win" on architecture — they're all Transformers. The differences a user feels come from training data, model size, the alignment/RLHF recipe, the surrounding tools (web, code, agents), and how open the weights are. Pick by task and constraints, not brand.
The Transformer — In the Order It Runs
GPT-2 forward pass reassembled as one continuous pipeline. Your prompt enters at the top, a single tensor flows downward, and one token comes out the bottom.
The Transformer,
in the order it runs.
The same GPT-2 walkthrough as the interactive explainer — but reassembled as one continuous pipeline. Your prompt enters at the top, a single tensor flows downward, and one token comes out the bottom. Each stage shows what changes and the shape of what's moving.
The whole journey on one screen
The map. Everything below is a zoom-in on one of these boxes. The rail on the left is the single tensor; only its shape changes — never its identity — until the very last step turns it into a word.
Tokenization
Text → discrete token IDs
The model can't read characters. The prompt is first chopped into tokens — whole words or word-pieces drawn from a fixed vocabulary of 50,257 entries. Common words map to one token; rarer ones split. Here "empowers" becomes two pieces, so six words yield six tokens.
Figure 1. Each box is one token; the number below is its ID — a row index into the embedding table built next.
Token embedding
Each ID → a 768-number vector of meaning
Each token ID is used to pull one row out of a giant learned lookup table of shape (50257, 768) — about 39M parameters. The row is a 768-dimensional vector. Tokens used in similar ways sit close together in this space, which is how raw IDs gain semantic meaning.
Figure 2. The ID is just a row number. Six tokens become a (6, 768) matrix — six rows of 768 numbers.
Positional encoding
Stamp each token with where it sits
The token vectors alone carry no order — “users empowers” would look identical to “empowers users.” A second learned table maps each position 0,1,2,… to its own 768-vector, so the model can tell first from last.
Figure 3. One position vector per slot. GPT-2 learns these during training rather than using a fixed formula.
Final embedding
token vector + position vector = the input to the stack
Add the two vectors element-wise. Each token now carries both what it means and where it is, in a single (6, 768) matrix. This is the tensor that enters the transformer blocks.
Figure 4. The end of "Embedding." From here the shape (6, 768) stays fixed all the way through the 12 blocks.
Transformer block — the loop that repeats 12×
attention mixes tokens together · MLP refines each one alone
This is the engine. The (6, 768) tensor passes through the block below; its output feeds straight into an identical block, twelve times. Two ideas do all the work:
Figure 5. One block. Stages 5a–5e below open up the two blue boxes — attention first, then MLP.
5a · Build Query, Key, Value
Inside attention, each token's vector is multiplied by three learned weight matrices to produce three new vectors. The classic search analogy: Query = what this token is looking for, Key = what each token offers, Value = the content it will hand over if matched.
Figure 5a. One embedding → three projections via learned weights. Q · K · V, each (6, 768).
Why three matrices? A token is one vector, but attention needs it to play three roles at once: ask a question, answer others' questions, and carry content. Reusing the same vector for all three would force "what I'm looking for" and "what I offer" to share directions. Three separate learned projections give three independent views of the same token.
What each output number is. Row i of Q is token i's query; its entry m is a dot product of the token's embedding with column m of WQ:
In GPT-2 the three matrices are stored fused as one 768 × 2304 weight (c_attn) and the result is sliced into Q | K | V; a bias is added to each.
5b · Split into 12 heads
Each of Q, K, V is sliced along its 768 columns into 12 heads of width 64. Every head runs attention independently and can specialise — one tracks grammar, another long-range meaning.
Figure 5b. 768 = 12 heads × 64. Same split applies to K and V.
5c · Masked self-attention (per head)
This is the heart of the model. Inside each head:
Figure 5c. The four moves of attention. The red triangle is the causal mask — the single rule that makes generation possible.
Everything below happens inside one head, on its own (6×64) slices of Q, K, V, with dk = 64.
① Score — the raw dot product. Multiply every query by every key. Entry (i,j) measures how aligned token i's query is with token j's key — one number summarising 64 multiply-adds:
② Scale — divide by √dk. Summing 64 products inflates the variance of S to about 64, so raw scores swing wildly and push softmax into a near one-hot spike where gradients vanish. Dividing by √64 = 8 pulls the variance back to ~1.
③ Mask — block the future. Add a matrix that is 0 on/below the diagonal and −∞ above it, so token i may attend to j only when j ≤ i. This one rule is what makes left-to-right generation valid.
④ Softmax — scores become weights. Exponentiate each row and normalise so it sums to 1. The −∞ cells become exp(−∞) = 0, so masked positions get exactly zero weight. Row i now reads "how much token i listens to each earlier token."
The weights say who to listen to; the Values say what they carry. Each token's output is the weighted average of the Value vectors it attended to — context folded into one vector.
Because row i's weights are zero past position i, oi can only ever be built from the current token and its left context.
5d · Recombine the heads
The 12 head outputs (each 6×64) are concatenated back into (6, 768) and passed through one more learned linear layer that lets the heads' findings mix. Attention is done.
Figure 5d. Heads merge back to the familiar (6, 768) shape, then a residual add returns the result to the spine.
Why a final matrix? Each head produced a (6×64) view in its own subspace. Concatenation restores (6×768) but leaves those subspaces siloed; the output projection WO lets the heads' findings interact and re-mix into one coherent update.
5e · MLP — refine each token
The second half of the block. A two-layer network expands each token vector from 768 to 3,072, applies a GELU nonlinearity, then compresses back to 768. No token sees another here — it's pure per-token refinement.
Figure 5e. Expand-then-compress. The wide middle layer is where most of the model's "knowledge" parameters live. Block output goes back to 05 — eleven more times.
Why it exists. Attention can only form weighted averages of existing vectors — it moves information between tokens but cannot compute new nonlinear features within a token. The MLP supplies exactly that: a per-token function with the capacity to detect and store patterns. Most of the model's factual "knowledge" lives here.
Project up to 3,072 dimensions (room for many feature detectors), apply a nonlinearity, project back to 768.
GELU(x) = x·Φ(x) ≈ 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)]) — a smooth gate that passes useful activations and softly suppresses the rest.
The wide middle holds the bulk of the block's parameters (~4.7M of ~7M). Width is capacity: more directions to recognise and recombine.
5f · The operators that wrap every sublayer
Attention and the MLP are never used bare. Each is wrapped in the same envelope — normalise → sublayer → dropout → add back. GPT-2 puts the LayerNorm before the sublayer (pre-norm):
x ← x + Dropout( MLP( LN2(x) ) )
Add the sublayer's input back to its output, so each sublayer learns a correction (a delta), never a full replacement.
It builds a gradient highway: since ∂(x+f(x))/∂x = I + ∂f/∂x, gradients reach early layers through the identity term — the reason 12 (or 96) stacked blocks can train at all.
Plain element-wise addition of two (6×768) tensors. No parameters, negligible cost.
Computed per token across its 768 features (μ and σ are scalars for that token), then rescaled by learned γ, β.
Keeps the scale of activations entering attention/MLP stable no matter how large the residual stream has grown — smoother, faster optimization.
μ, σ² are the mean and variance of the token's 768 numbers; ε (~1e−5) guards the divide; γ, β let the model rescale or even undo the normalization.
During training only, randomly zero a fraction p of activations (GPT-2: p = 0.1) on attention weights, the MLP, and residual outputs.
Stops the network over-relying on any single path; forces redundant, robust features — akin to averaging many thinned sub-networks.
Survivors are divided by (1−p) so the expected sum is unchanged; at inference dropout is off and nothing is scaled.
Output projection → logits
From 768 numbers to one score per vocabulary word
Only the last token's vector matters for predicting what comes next. A final linear layer projects its 768 numbers up to 50,257 — one raw score (a logit) for every possible token in the vocabulary.
Figure 6. Higher logit = the model thinks that token is a more likely continuation.
This final linear layer is the unembedding. In GPT-2 its weights are tied to the token-embedding matrix E — the same table that turned tokens into vectors at the start now turns the final vector back into per-token scores.
Softmax → probabilities → sampling
Turn scores into odds, then pick a winner
Softmax squashes the 50,257 logits into probabilities that sum to 1. Three dials shape the final pick:
Figure 7. A probability for every word. The dials below decide how this distribution is read.
Softmax exponentiates every logit and divides by the total, so each probability is positive and all 50,257 sum to 1. Because of the exponential, a modest gap in logits becomes a large gap in probability.
Worked from the chart. If the leading logits exponentiate to relative weights 41 : 22 : 13 : …, dividing each by the running total over all 50,257 tokens gives the bars above:
T divides the logits first. T<1 widens the gaps (sharper, safer); T>1 shrinks them (flatter, riskier); T→0 becomes pure argmax.
Keep only the top k tokens, or the smallest set whose probabilities reach p; everything else is dropped to zero.
The surviving probabilities are rescaled to sum to 1 again, then one token is sampled from that trimmed distribution.
dialTemperature
Divides logits before softmax. <1 sharpens (safe, repetitive); >1 flattens (creative, risky); =1 leaves it unchanged.
dialTop-k
Keep only the k highest-probability tokens as candidates; discard the long tail entirely.
dialTop-p
Keep the smallest set of tokens whose probabilities add up to p — an adaptive cutoff that widens or narrows with confidence.
Emit token, then loop
Autoregression — one word at a time
One token is sampled — say "create". It's appended to the prompt, and the entire pipeline runs again on the now-longer sequence to produce the next word. This repeat-until-done loop is how a few hundred million fixed numbers write fluent text.
Figure 8. The output becomes part of the next input. Stages 01–07 repeat for every single word the model writes.
The quiet helpers (present throughout, not shown on the spine)
stabilityLayer Normalization
Rescales each token's vector to a consistent mean and variance before attention and before the MLP. Keeps training stable and fast.
regularizationDropout
Randomly zeroes activations during training only so the model doesn't over-rely on any one path. Off at inference.
gradient flowResidual Connections
The "+" shortcuts around attention and the MLP let signals (and gradients) skip layers — the trick that makes 12 deep blocks trainable.