A checklist for text normalization
Normalization bugs are dull and expensive: two strings that look identical on screen do not compare equal, and a deduplication step silently keeps both. This is the order I apply.
- Decode strictly. Fail on invalid bytes instead of replacing them, so broken sources are noticed.
- Unicode form. NFC for text that people will read, NFKC when matching matters more than appearance.
- Invisible characters. Remove zero-width spaces, soft hyphens and byte-order marks.
- Whitespace. Map non-breaking and other exotic spaces to a plain space, then collapse runs.
- Quotes and dashes. Unify only if the downstream model was trained on unified text.
- Case. Lowercase last and only where the task allows it; casing carries signal for entities.
import unicodedata, re
def normalize(s: str) -> str:
s = unicodedata.normalize("NFC", s)
s = re.sub(r"[\u200b\u200c\u200d\u00ad\ufeff]", "", s)
s = re.sub(r"\s+", " ", s)
return s.strip()
Whatever the pipeline does, apply the same function at training and inference time and keep it under test. A dozen awkward strings in a unit test catch most regressions.