What word vectors capture

1A word's vector and the closest words

The vectors of this note come from the skip-gram with negative sampling of the note on word2vec, trained on the book: d = 100 numbers per word, a radius drawn from 1 to m = 5 for each target, and k = 5 drawn words per pair, with β = ¾. As the original program does, it gives no vector to the words seen fewer than five times, and it skips frequent words such as the at random, more often the more frequent they are, since they stand next to every word and teach little. A book is small next to the corpora word2vec was made for, so training read it 20 times, where the program reads a text 5 times unless told otherwise. Training starts from random numbers, so it was run five times, from five random starts: the figures and the numbers quoted show the first run, and the text claims only what held in at least four of the five.

A bag of words describes a text, as in the notes on linear models and on word2vec: x ∈ ℝ^V, one coordinate per word of the vocabulary, named by its word and counting how often the text uses it, so almost all of them are 0; a single word alone is a one-hot vector. Here V = 4,993, the words that have a vector. The book has 14,921 different words, and the other 9,928, two thirds of them but only 4.3% of the text, get none.

An embedding describes a word: v_w ∈ ℝ^d, its row of W_in, about fifty times shorter. It is dense: the vector of curate has 51 negative numbers, 49 positive ones and no zeros. And it is learned, so none of its numbers has a name. Turning every row of W_in and every column of W_out by the same rotation would change every number and leave every score u_c^Tv_o as it was, so training has no reason to prefer one set of axes over another. What the numbers hold is in how the vectors lie against each other, and the cosine of the note on the vector space model measures it, through the angle θ between two vectors:

(θ is the angle here; the logistic function has been σ since the note on word2vec). Two vectors that point the same way have cos θ near 1, and two at a right angle have 0. Unlike the tf-idf weights of the note on the vector space model, which are never negative, the numbers of an embedding can be negative, so the cosine runs from −1 to 1, although few words point away: only 16 of the other 4,992 sit more than 90° from curate. The cosine sets length aside, and length is a poor guide here, because it depends largely on how often a word appears: words seen 5 to 9 times have a median length of 1.60, and words seen 50 to 199 times, 2.61.

The figure draws the picked word's vector and the vector of its closest word in the plane the two of them span, where their angle is exact, and around them it counts every other word of the vocabulary by its angle from the picked word, degree by degree. Pick a word, and switch from its embedding to its bag of words.

Two words picked at random are usually far apart: half of all the pairs of the vocabulary sit more than 66° apart, and the closest words of a word are a thin edge of the crowd, ahead of the rest. As bags of words, every word is at 90° from every other, since two different one-hot vectors share no coordinate. The helmet sits by the basin that Don Quixote takes for Mambrino's helmet, and the giants by the windmills he takes for giants.

Close means related, not alike. The closest word to bread is cheese, and to summer it is winter, in all five runs: opposites are used in the same places, in summer and in winter. And each spelling has one vector, whatever its senses, like the senses of light in the note on WordNet; section 5 shows what that costs.

2Words that fill the same blanks

Training never pulls two targets toward each other. Each pair (o, c) moves v_o toward u_c, a row of W_in toward a column of W_out, as in the figure of negative sampling of the note on word2vec. Even curate and barber, which stand within five words of each other 42 times, are each pulled toward the other's context vector u, never toward the other's row. Two rows end up close when they are pulled toward the same context vectors: when the two words fill the same blanks.

The book uses quest 46 times and search 45 times, and both in the same frames, such as in quest of adventures and in search of adventures. The two never stand within 20 words of each other, yet quest is the closest word to search in all five runs, about 40° away. Through W_out, the model expects adventures near both: σ(u_adventures^Tv_quest) is 0.88 and σ(u_adventures^Tv_search) is 0.75.

Why does the direction carry this, and not the length? A score splits into two lengths and an angle,

so it rises either by turning the two vectors toward each other or by lengthening them. The loss of the note on word2vec makes −log σ(u_c^Tv_o) small for an observed pair, which takes a large score. Lengthening v_o scales all its scores at once, the drawn words' as well, so it pays only once v_o points toward the words it was seen with and away from the drawn ones. Without drawn words, nothing would stop every vector from growing long in one shared direction, as Goldberg and Levy (2014) point out: the drawn words make the direction matter, and the direction is what section 1's cosine measures. The lengths do grow too, to between 1.18 and 4.48 in the book's vectors.

The figure follows a designed example with vectors of two numbers, as in the note on word2vec: two targets, quest and search, both carmine; a context word that both are seen with, adventures, in green; and one word drawn from the book for each target, combat and ought. Take steps of gradient descent with η = 1.2, then switch to a version where search is seen instead with a word of its own, finding.

Words with similar neighbors end up with similar vectors, as the note on word2vec promised. Synonyms seldom stand side by side, since a writer uses one or the other, so this indirect route, through the words around them, is how they meet. What training asks for is each v_o aligned with its u_c; nothing in it asks for two targets to be close, which happens because they share context vectors.

3The odd one out

If related words point the same way, a list should show which of its words does not belong. Leave out each word in turn and add up the vectors of the others: the sum is long when the words left in point the same way, so the longest sum is the one without the intruder. Written with argmax, “the w that makes this largest”:

where L is the list and every vector is first scaled to length 1. For three words w, w′ and w″ with vectors of length 1, the length of their sum depends only on their cosines,

so the sum is long only when its words' cosines are all large, and a word at a wide angle from the others brings a small cosine into every pair it forms. The vectors are scaled first because, as trained, a short vector takes little away from the sum when it is left out, and the rarest words have the shortest vectors: the rule would tend to pick the rarest word. On vectors of length 1 it is the rule of the first word2vec paper (Mikolov, Chen, Corrado and Dean 2013), which picks the word farthest from the list's average. Build a list of three to six of the sixteen words below, and compare the vectors scaled to length 1 with the vectors as trained.

Every list of three words from one of the figure's four groups (the mounts, the foods, the people, the island's government) and one word from another picks the intruder when the vectors are scaled, in all five runs, but only 83% of them do as trained. The rule reads only the angles: cheese and onions end up together because the book uses them in the same places, not because a resource such as WordNet files both under food. It fails where the book uses in the same way words that a reader would set apart: in the list sancho, quixote, dulcinea, teresa, rocinante, four people and a horse, the three longest sums lie within a few hundredths of each other, and which one wins changes from run to run.

4Analogies

Do directions mean anything? An analogy a : a* :: b : b*, read “a is to a* as b is to b*”, asks for the word that stands to b as a* stands to a: knight is to knights as squire is to squires. If the step from a to a* is a direction in the space, adding it to b should land near b*:

again on vectors of length 1. Writing t for v_b − v_a + v_a*, the cosine splits into the cosines of w with the three given words,

so the answer must be close to b and to a*, and far from a. The three given words are left out of the search, as the first word2vec paper leaves them out: b itself has cos θ_bb = 1 in that sum, and over 25 analogies tried on this book, a search over every word puts one of the given words first in 22. Mikolov, Yih and Zweig (2013) found such regularities in the vectors of a recurrent network, a network that reads a text one word at a time, which a later note covers, and word2vec made them widely known.

The two steps are only partly the same vector: the cosine between the step from knight to knights and the step from squire to squires is 0.68. Part of every answer comes from staying near b, and the figure's second mode shows how much: the words closest to b alone. Pick an analogy, and compare.

The plural comes first in all five runs, and squires, only the 408th closest word to squire, owes its place to the step. Sancho comes first too, but he was already the closest word to quixote: there the step adds little. What one book gets right is its own world and its grammar. Relations it shows seldom, such as son to daughter and brother to sister, need far more text, which is where section 5 goes.

5Pretrained vectors

Training on billions of words is costly, so a group trains once and publishes the result, and anyone can download it: a file with a line per word, the word and its numbers. fastText's vectors for English and for Spanish (Grave et al. 2018) hold 2,000,000 words each, with 300 numbers per word, trained on the web pages of Common Crawl and on Wikipedia. Their model is CBOW, the word2vec model that guesses a word from its neighbors, with pieces of words added, which the next note explains. The file holds a row per word, taken from the model's input side: for word2vec it would be the word's row of W_in, while fastText averages that row with the rows of the word's pieces. Here V = 2,000,000 and d = 300. Training one's own vectors remains an option, as this note did on the book.

The table has 4,839 of the book's 4,993 words in lowercase; the 154 it lacks are mostly names, such as rocinante and cardenio, and words with an apostrophe, such as quixote's. In the figure, the closest words of the web's vectors are searched among the table's 100,000 most frequent lowercase words. Pick a word, and compare its closest words on the book with its closest words on the web.

On the web, curate is mostly a verb, to select and present, and giants stand with behemoths and titans, far from any windmill. One spelling keeps one vector, and the corpus decides which sense fills it. Relations follow the corpus too. The web's vectors complete son : daughter :: brother : sister, where the book's five runs put sister between 5th and 35th, and son : daughter :: nephew : niece, where the book has no vector for nephew at all. The book completes king : kingdom :: governor : island, Sancho's governorship, in four of its five runs, and the web's vectors put island 158th.

6Pooling: one vector for a text

A classifier, such as those of the note on linear models, reads one vector per text. With embeddings, a text of N words has N vectors, and pooling combines them into one:

where e(w) = v_w is the embedding of the word w, its row of W_in, as trained. The usual pools are the mean and the maximum of each of the d numbers:

The sum works as well: it is N times the mean, so it gives the same cosines; which pool fits depends on the task. x is again the text's vector, as the bag of words of section 1 was, but with d numbers instead of V, and W_in joins the two: the sum of a text's embeddings is W_in^Tx for its bag of words x, the select step of the note on word2vec fed a whole bag of words instead of a one-hot vector. Under the weights of the vector space model, a query and a text with no word in common score 0; their pooled embeddings may still be close. In the figure, pick a pair of short texts, and compare their mean with their maximum.

A classifier then reads x. One with a hidden layer, which the note on word2vec defined, passes it through a layer of weights W and the logistic function, applied to each number, before a softmax like the one in the note on linear models, here with weights U, a row per class:

with a 1 at the start of x, so that the first column of W holds the biases, as in the note on linear models, which had no hidden layer. Pooling loses the order again, as the bag of words did: Sancho's line and Don Quixote's reading pool to the same vector. Frequent words such as the weigh on the mean unless they are dropped, as stopwords were in the note on the vector space model, or weighted down. Later notes keep the order, with recurrent networks and attention, and pool vectors that depend on the sentence.

7Summary

The note's ideas in NumPy. w_in holds the embeddings, a row per word of the vocabulary, and unit the same rows scaled to length 1, unit = w_in / np.linalg.norm(w_in, axis=1, keepdims=True); w_out holds the context vectors, a column per word; w, w2, o, c, a, a_star and b are words' positions in the vocabulary, L those of a list's words and ids those of a text's words; W and U are a classifier's weights.

The note's data: the web's vectors are fastText's English and Spanish tables, cc.en.300 and cc.es.300 (Grave et al. 2018), under the Creative Commons Attribution-ShareAlike 3.0 license, from fasttext.cc; the page shows only what was computed from them, such as closest words, cosines, dot products and lengths, never the vectors themselves. The book's vectors were trained for this note.