Chapter 2 — Working with Text Data
How raw text becomes tokens, token IDs, and embedding vectors — the entire preparation pipeline that feeds an LLM.
Before an LLM can do anything interesting, raw text has to be turned into numbers it can actually crunch. Chapter 2 is the gap between "I have a text file" and "I have a tensor ready to feed into the model". It's the necessary bridge between the bird's-eye view from Chapter 1 and the actual PyTorch code we'll write in Chapter 3.
What you'll learn
By the end of this chapter you'll be able to answer, in plain language:
- Why neural networks can't read text directly and why we need embeddings.
- How to tokenize text into words and punctuation that the model can work with.
- How tokens are converted into integer token IDs via a vocabulary.
- The role of special tokens like
<|unk|>and<|endoftext|>. - What byte pair encoding (BPE) is and why GPT-style LLMs use it.
- How to build input-target pairs for next-word prediction using a sliding window.
- How token IDs become embedding vectors the model can train on.
- How positional embeddings teach the model about word order.
Chapter contents
2.1 — Understanding word embeddings
Why we need to convert text into continuous-valued vectors before anything else can happen.
2.2 — Tokenizing text
Splitting a passage into individual words and punctuation characters.
2.3 — Converting tokens into token IDs
Building a vocabulary and mapping each token to a unique integer.
2.4 — Adding special context tokens
Handling unknown words and marking boundaries between unrelated texts.
2.5 — Byte pair encoding (BPE)
The subword tokenization scheme GPT-2, GPT-3, and friends actually use.
2.6 — Data sampling with a sliding window
Turning one long text into many input-target pairs for next-word training.
2.7 — Creating token embeddings
Embedding layers as a learnable lookup table from token IDs to vectors.
2.8 — Encoding word positions
Adding absolute positional embeddings so the model knows token order.
2.9 — Summary
The one-page recap of the entire text-preparation pipeline.
A heads-up: this chapter is mostly mechanical — text in → tokens out → numbers out. The interesting learning arrives in the next chapter, where we wire those numbers into a model that actually pays attention to them.