How does subword tokenization such as byte pair encoding work?
Byte pair encoding starts from small units like characters or bytes and repeatedly merges the most frequent adjacent pair into a new token, until a target vocabulary size is reached. Common words become single tokens, rare words split into pieces.
Explain it simply
Think of how people shorten WhatsApp messages. Frequent phrases such as "by the way" become "btw", while rare words stay spelled out. Byte pair encoding builds its vocabulary in a similar frequency-driven way.
Start with every character as its own token. Count which adjacent pair appears most often in the training text, merge that pair into a new token, and repeat. After thousands of merges, "the" and "ing" are single tokens, and a word like "unbelievably" may split into "un", "believ", "ably".
The benefit is a fixed, manageable vocabulary with no truly unknown words: anything can fall back to smaller pieces, even single bytes. The cost is that unusual words, names and other languages may be chopped into many tokens.
Example
Corpus words: low, lower, lowest
Start: l o w | l o w e r | l o w e s t
Merge 1: (l,o) -> lo
Merge 2: (lo,w) -> low
Result: "low" is one token, "er" and "est" become separate piecesWhat to say in the interview
"BPE learns merges from corpus statistics: repeatedly join the most frequent adjacent pair until the vocabulary hits a chosen size. It balances vocabulary size against sequence length and avoids out-of-vocabulary words by falling back to subwords or bytes."
Common mistakes
- Thinking tokens are always whole words.