My NLP Notes

Scoring named-entity recognition without fooling yourself

2026-06-21

Most tokens in running text are outside any entity, so a tagger that predicts O everywhere can reach 90% token accuracy. The useful measure is entity-level F1: a prediction counts only when both the span and the type match the reference exactly.

Three ways to count

ModeA match requiresUse it for
StrictSame boundaries and same typeHeadline number, comparison with papers
Type onlyOverlap and same typeFinding boundary disagreements
Boundary onlySame boundaries, any typeFinding confusable classes

Things that distort the score

I report strict micro F1, a per-type table, and the unseen-entity score. Together they fit in a few lines and answer most follow-up questions in advance.

← All notes