My NLP Notes

Byte-pair encoding, worked through by hand

2026-09-14

Word-level vocabularies break on the first unseen word, and character-level ones make sequences very long. Byte-pair encoding sits between the two: start from characters and repeatedly merge the most frequent adjacent pair until the vocabulary reaches the size you want.

A tiny corpus

Take the counts low:5, lower:2, newest:6, widest:3. Split every word into characters plus an end-of-word mark. The most frequent pair is e s (9 times), so it becomes es. Next comes es t, then est </w>, and the suffix -est has become a single token after three merges.

merges = [("e","s"), ("es","t"), ("est","</w>"), ("l","o"), ("lo","w")]

What to remember

I keep a short script that prints the merge table step by step. Watching the first fifty merges on a new corpus is a quick way to spot encoding problems and stray markup.

← All notes