Words and Tokens¶
Jurafsky, D., & Martin, J. H. (2026). Words and Tokens. Speech and Language Processing.
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Type–Token Distinction
- Segment by character rather than by word and the same string yields seventeen letter occurrences over nine forms, or twenty-two over ten if the spaces are counted too.
This sourceStates that whether a capitalized and an uncapitalized string count as the same word type depends on the task, so the type count over a fixed text moves with the criterion while the count of running words does not.
- Segment by character rather than by word and the same string yields seventeen letter occurrences over nine forms, or twenty-two over ten if the spaces are counted too.
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Registry ID ref:35198717d21f · see in the full table