01The big idea
It writes one piece at a time.
A language model is the engine inside a chatbot. It never plans a whole answer. It guesses the next small piece of text, adds it, and guesses again.
The text so far
The best thing about rainy days is the sound of rain on the window.
02Tokens
Text becomes tokens.
Before the model reads anything, the text is cut into tokens. A token is a common word, a piece of a longer word, or a single character such as a comma.
Simplified tokenizer for illustration. It always takes the longest piece it knows from a list of 300 words and word pieces, plus single characters. Real tokenizers use much larger lists learned from data.
characters per token in ordinary English, or about three quarters of a word. So 100 tokens is roughly 75 words.
03Embeddings
Tokens become numbers.
Each token id is swapped for a list of numbers called an embedding. Tokens that get used in similar ways end up with similar lists, so they sit close together.
below zeroabove zero16 made-up numbers per word
Real models use far longer lists. GPT-3, published in 2020, gives every token 12,288 numbers.
04Attention
Words look at each other.
A word means more once you know its neighbours. In a step called attention, every token checks the other tokens and pulls in information from the ones that matter most to it.
Dot area shows the weight. Each row shares out 100%. Chatbot models let a word look only at earlier words. This demo lets words look both ways to keep the picture simple.
05Layers
Layers sharpen the guess.
The numbers flow up through a stack of layers. Each layer runs attention, then a small calculation on every token, and passes the result up to the next layer.
attention: tokens share informationcalculation: each token worked on alone
Best guess for the word after “rain on the”
GPT-3 has 96 layers and 175 billion parameters. Parameters are the adjustable numbers inside the layers. Training is what sets them.
06Probabilities and sampling
Choosing the next word.
Every step ends with a score for each possible token. A setting called temperature turns those scores into chances, and then one token is drawn at random using those chances.
Entropy measures how unpredictable the choice is. 0 bits means one certain answer. 3 bits means all eight words are equally likely.
07Training
How it learned.
Training repeats one small exercise billions of times. Hide the next word of real text, let the model guess, measure how wrong the guess was, and nudge the numbers inside so the right word becomes a little more likely.
The cat sat on the ?
Model guessAfter this first training, people rate the model's answers, and the model is tuned toward the answers they prefer. That tuning is what makes it follow instructions and answer helpfully.
GPT-3 was trained on about 300 billion tokens of text. The counter above is scaled to that total.
08The whole chain
Not magic. Prediction at enormous scale.
- Text
- Tokens
- Numbers
- Attention
- Layers
- Probabilities
- Choice
The chosen token joins the text, and the whole chain runs again for the next one.
The model picks words that are likely to come next. Nothing in that process checks whether they are true, so check important facts against a source you trust.
Sources: OpenAI Help Center, “What are tokens and how to count them?” (help.openai.com). Brown et al., “Language Models are Few-Shot Learners”, 2020 (arXiv 2005.14165), for GPT-3’s layers, parameters, embedding size and training tokens. Vaswani et al., “Attention Is All You Need”, 2017 (arXiv 1706.03762), which introduced the transformer design. All other numbers are illustrative models.
Made by Jaw Dropping Presentations