Subword Tokenization (BPE & WordPiece)
Subword tokenization is how transformers handle huge vocabularies and unseen words, and understanding it demystifies model inputs.
Learn it
Subword algorithms build a vocabulary of frequent character sequences so any word is expressible as known pieces. You will contrast BPE (merge most frequent pair) with WordPiece (merge to maximize corpus likelihood) and Unigram, and train a tokenizer with the Hugging Face tokenizers library.
By the end you’ll be able to
- Explain why transformers use subword rather than word tokens
- Compare BPE, WordPiece, Unigram; train a tokenizer
Free resources
Practice
Train a BPE tokenizer on a small text corpus and inspect how it splits rare and made-up words.
Build it — your output
A notebook that trains a subword tokenizer and shows token splits for 5 tricky words.
Prove it — check yourself
You’re ready to move on when you can honestly say yes to each:
- Explain how subword tokens solve out-of-vocabulary words
- State the key difference between BPE and WordPiece merges
- Read a tokenizer's output back into text
Do it with AI
Use a free AI tool as your tutor and pair-programmer. Copy a prompt below into Claude, ChatGPT, or Gemini (for building, try Cursor or Copilot).
The habit: Try it yourself first — a little struggle is where the learning actually happens. Then ask for a hint before the answer — never copy blindly.
- Understand itClaude / ChatGPT / Gemini
I'm a beginner learning "Subword Tokenization (BPE & WordPiece)". Explain it in simple terms with a real-world analogy, then ask me ONE question to check my understanding — don't reveal the answer.
- PracticeClaude / ChatGPT / Gemini
Give me 3 short practice problems on "Subword Tokenization (BPE & WordPiece)" (my goal: explain why transformers use subword rather than word tokens), from easy to hard. Don't include the solutions yet — I'll try them first, then ask you to check.
- Get unstuckClaude / ChatGPT / Gemini
I'm stuck on this exercise about "Subword Tokenization (BPE & WordPiece)": [paste your attempt here]. Give me ONE small hint to move forward — do NOT give me the full solution.
- Build & reviewCursor / Copilot
Here is my work for a "Subword Tokenization (BPE & WordPiece)" task: [paste your code or answer]. Review it: point out what's wrong and WHY, and suggest improvements — but let me make the fixes myself.
- Prove itClaude / ChatGPT / Gemini
Quiz me on "Subword Tokenization (BPE & WordPiece)". Ask me 5 questions one at a time, wait for my answer each time, then tell me if I'm right and briefly why.