My NLP Notes

TF-IDF is still the baseline to beat

2026-08-02

Before fine-tuning anything I fit a TF-IDF vectorizer and a linear classifier. It trains in seconds, needs no GPU, and gives a number that every later experiment has to justify itself against.

Settings that matter

SettingUsual choiceWhy
n-gramswords 1–2, characters 3–5Character n-grams absorb typos and inflection
Sublinear TFonA word repeated 50 times is not 50 times as important
Minimum document frequency2–5Drops one-off noise and shrinks the matrix
Classifierlogistic regression or linear SVMBoth cope well with very wide sparse input

Reading the result

If the baseline is already close to the target metric, the task is mostly lexical and a large model buys little. If it is far off, look at the errors first: label noise and duplicated examples between train and test explain more surprising scores than model choice does.

The feature weights are also a free debugging tool. The top positive terms for each class should make sense to a person who knows the domain. When they are dates, signatures or template text, the dataset has a leak.

← All notes