TF-IDF is still the baseline to beat
Before fine-tuning anything I fit a TF-IDF vectorizer and a linear classifier. It trains in seconds, needs no GPU, and gives a number that every later experiment has to justify itself against.
Settings that matter
| Setting | Usual choice | Why |
|---|---|---|
| n-grams | words 1–2, characters 3–5 | Character n-grams absorb typos and inflection |
| Sublinear TF | on | A word repeated 50 times is not 50 times as important |
| Minimum document frequency | 2–5 | Drops one-off noise and shrinks the matrix |
| Classifier | logistic regression or linear SVM | Both cope well with very wide sparse input |
Reading the result
If the baseline is already close to the target metric, the task is mostly lexical and a large model buys little. If it is far off, look at the errors first: label noise and duplicated examples between train and test explain more surprising scores than model choice does.
The feature weights are also a free debugging tool. The top positive terms for each class should make sense to a person who knows the domain. When they are dates, signatures or template text, the dataset has a leak.