Notes on language processing
- Byte-pair encoding, worked through by hand
2026-09-14 · Why subword vocabularies exist and what the merge table actually stores.
- TF-IDF is still the baseline to beat
2026-08-02 · A sparse model takes ten minutes to build and tells you how hard the task is.
- Scoring named-entity recognition without fooling yourself
2026-06-21 · Token accuracy looks great and means little. Count entities instead.
- A checklist for text normalization
2026-05-09 · Unicode forms, whitespace, quotes and the order to apply them in.