Words, lemmas and WordNet

1What a word is made of

Linguistics describes language at four levels, each with the unit it studies and the discipline that studies it:

An NLP system may need any of them; this note covers the first three, starting with the word. Its smallest piece with both a form and a meaning is the moneme, and there are two kinds. Speaking holds one of each: speak, the lexeme, carries the meaning and is shared with related words such as speaker; -ing, a morpheme, adds grammatical meaning, here an action in progress. Morphemes mark gender, number, person, tense and the like.

Words split into two groups by whether they take such morphemes. Variable words, such as nouns, adjectives, verbs and pronouns, change their form; invariable ones, such as prepositions, conjunctions and interjections, keep a single form. Variable words are what makes a vocabulary large. Pick a word with the buttons to see all its forms in the book, each split into its lexeme, in ink, and its ending, in carmine, with its count. Below are the forms of the same word in the Spanish original.

Spanish puts person, number, tense and mood into its verbs' endings, where English uses separate words: diremos is we will say. A count takes the Spanish form for one more word, and the English words for three it already has; section 2 measures how much of the gap between the books that explains.

2Stems and lemmas

Two methods gather a word's forms. Stemming cuts endings off by rules, with no dictionary: the Snowball stemmers, which grew out of Martin Porter's 1980 stemmer for English and now cover many languages, turn speaking and speaks into speak. What remains, the stem, need not be a word. Lemmatization returns each form's lemma, the form a dictionary lists it under: said and says both give say. A lemmatizer never invents a word, but it needs each form's part of speech, since saw is a form of see as a verb and a word of its own as a noun; without it, NLTK's lemmatizer takes every word for a noun.

Pick a word to see which of the book's words share its stem and which share its lemma; a word in only one of the two groups is outlined in carmine. Below, how far each method shrinks each book's vocabulary.

Each method fails in its own way, as the examples show, but lemmatizing brings the books close: the translation's 14,921 different words become 10,337 lemmas, and the original's 22,605 become 12,596. A word the list lacks counts as its own lemma. That is right for most of them, since the list leaves out the words that are lemmas already, such as with, but the Spanish list also lacks verbs with a pronoun attached, such as decirle (to tell him), and the old spellings of a book from 1605, such as agora for ahora (now), so the original's true count is lower still.

3Parts of speech and names

Sentences have parts too. Their words form groups, the constituents, and smaller groups sit inside larger ones. In The squire rides a donkey, the squire is a noun phrase (NP): he can take its place, as in He rides a donkey, which shows that the two words work as one. Rides a donkey is a verb phrase (VP), a verb with a noun phrase of its own.

Parsers recover this structure, often starting from each word's category, its part of speech: noun, verb, preposition and so on. Labeling every word with it is part-of-speech tagging (POS tagging). Named-entity recognition (NER) reads the same words with another question: not what kind of word each one is, but what it refers to, such as a person, a place, an organization or a date.

The figure shows the first line of chapter I and the chapter's title, each word with its part of speech in the tags of Universal Dependencies, a set used across languages, and the named entities underlined in carmine. Pick a word to read its tags, and compare Mancha in the two.

Mancha is a proper noun both times, so its part of speech cannot tell a place from part of a name. The clues come from two sources. The form of a word, such as a capital letter, says it may be a name; its neighbors say what kind: after a village of comes a place, after Don a person. NER is called a morphosyntactic task for that reason: it needs both the word's form and its place in the sentence.

4Meanings in a network

Meanings are related too, and two relations matter most. A knight-errant is a kind of knight: knight-errant is a hyponym of knight, and knight a hypernym of knight-errant. A helmet is a piece of a suit of armor: helmet is a meronym of suit of armor, and suit of armor a holonym of helmet. The first relation sorts things into classes; the second takes them apart.

WordNet records both for English, and it is organized by meanings rather than by words: each synset is a set of synonyms with one meaning, a definition, and links to other synsets. George Miller's team at Princeton began building it by hand in 1985; version 3.0 has 117,659 synsets and 147,306 different words and phrases. NLTK includes it, and its lemmatizer is built on it: an irregular form such as saw is looked up in a list of exceptions, and a regular one loses its ending only if what remains is in WordNet. BabelNet goes further: it merges WordNet with Wikipedia and other resources, so each of its entries gathers a concept's words in many languages, along with names of people and places that WordNet lacks.

The figure shows a small piece of WordNet, with the Spanish words that the Multilingual Central Repository gives each synset. In Kinds, the chain of hypernyms above knight runs up to entity, and picking a synset draws its chain in blue; in Parts, ten of the parts of a suit of armor run from head to foot. One of them, the thigh plate, is in Spanish a quijote.

5One spelling, several words

Synonymy is several forms for one meaning. The reverse, one form for several meanings, comes about in two ways. In polysemy, one word's senses grew apart from a shared origin, often by metaphor. In homonymy, two unrelated words share a spelling by chance. Dictionaries mark the difference: they give each homonym an entry of its own, and list the senses of a polysemous word inside its one entry. WordNet makes no such difference: it gives light 47 senses, 15 as a noun, 6 as a verb, 25 as an adjective and 1 as an adverb, with no mark of which ones share an origin.

The book's light shows both. Light as brightness comes from a root *leuk-, and light as not heavy from a different one, *legwh-: two words that look and sound alike by an accident of history. Each grew senses of its own. From brightness came a lamp, and the verb in to light a candle; from not heavy came to light upon something, which first meant to get down from a horse and so lighten its load, and to make light of it. The figure sorts the book's 98 uses of light by sense, labeled by hand; pick one to see its share and a line from the book.

Choosing a word's sense from its sentence is word-sense disambiguation, and a lemmatizer does not do it: light gets one lemma whatever it means. The word vectors of later notes share that blind spot, one vector per word, until encoders build each word's vector from its sentence.

6Summary

The note's ideas in NLTK, a Python library for working with text. tokens holds a sentence's words, and wn is nltk.corpus.wordnet, with the Open Multilingual Wordnet installed for Spanish.

The note's data: the lemmas come from Michal Měchura's lemmatization lists (ODbL), the stems from the Snowball stemmers, the synsets and their relations from Princeton University's WordNet 3.0, and their Spanish words from the Multilingual Central Repository 3.0 (CC BY 3.0).