Pro Pro lesson
Words as vectors
Turn words into lists of numbers, and meaning into distance, using real word vectors.
This one’s part of How Models Learn Pro — every lesson, every interactive, and new ones as they land. About 10 minutes.
Already Pro? Sign in and you’ll go straight in.
Inside “Words as vectors”
What you’ll learn
- Words into numbers
- Similar words point the same way
- A map of meaning
- Arithmetic with meaning
- Real models learn their own features
Key terms
- Vector
- An ordered list of numbers. It can be pictured as an arrow pointing somewhere in space.
- Embedding
- The vector a model uses to represent a word (or other item), arranged so that similar things get similar vectors.
- GloVe
- A widely used set of word vectors from Stanford (2014), learned from how often words appear near each other in billions of words of text.
- Dimension
- One position in a vector. A GloVe vector has 300 numbers, so it lives in a 300-dimensional space.
- Cosine similarity
- A score from −1 to 1 that measures the angle between two vectors: 1 means they point the same way.
- Nearest neighbours (of a word)
- The items whose vectors are most similar to a given item’s vector.
- Direction
- A line through the space, such as female − male, found by averaging the differences between word pairs. Measuring a vector along it gives a meaningful score.
- PCA
- Principal component analysis: a standard way to flatten many dimensions into a few by keeping the directions along which the data spreads out most.
- Word analogy
- Vector arithmetic such as king − man + woman, whose result lands near a related word like queen.
Sources
- GloVe: Global Vectors for Word Representation (Stanford NLP project page) — Source of the vectors: glove.6B, trained on Wikipedia 2014 + Gigaword 5 (6B tokens, 400K uncased vocabulary, up to 300 dimensions). Training makes dot products match log co-occurrence probabilities. PDDL licence.
- GloVe: Global Vectors for Word Representation (Pennington, Socher & Manning, 2014) — The original GloVe paper: a weighted least-squares model trained on global word–word co-occurrence counts, with a vocabulary of the 400,000 most frequent words; evaluated on word analogies.
- Linguistic Regularities in Continuous Space Word Representations (Mikolov, Yih & Zweig, 2013) — Showed that relations appear as constant vector offsets in learned word vectors: King − Man + Woman lands very close to Queen.
- Linguistic Regularities in Sparse and Explicit Word Representations (Levy & Goldberg, 2014) — Defines 3CosAdd, the analogy method used here: pick the word maximising cos(b*, b − a + a*) over the vocabulary excluding the question words.
- Speech and Language Processing (3rd ed. draft), Chapter 5: Embeddings (Jurafsky & Martin) — Distributional hypothesis (words in similar contexts have similar meanings), cosine similarity, the parallelogram analogy method and why the input words must be explicitly excluded; describes GloVe.
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Bolukbasi et al., 2016) — Finds a gender direction in word2vec vectors (trained on Google News) as the first principal component of ten pair differences such as she − he. We use the average of the differences instead, a close cousin.
- Principal component analysis: a review and recent developments (Jolliffe & Cadima, 2016) — PCA reduces many dimensions to a few new uncorrelated variables that successively maximise variance — the method used to flatten the vectors onto a 2-D map.