GloVe: Definition and Examples

GloVe, acronym for Global Vectors for Word Representation, is a word embedding model created in 2014 by Jeffrey Pennington, Richard Socher, and Christopher Manning at the Stanford NLP laboratory. Its goal is to transform each word in a vocabulary into a dense numerical vector that captures its semantic meaning.

Unlike Word2Vec, which learns representations from local context windows, GloVe exploits a global word co-occurrence matrix across the entire corpus. The algorithm factorizes this matrix to produce vectors where geometric relationships (distances, directions) between vectors reflect semantic relationships between words. For example, the famous vector analogy king - man + woman ≈ queen works remarkably well with GloVe.

The model is based on the intuition that the ratio of co-occurrence probabilities between two words relative to a third encodes rich semantic information. GloVe optimizes a cost function that preserves these ratios in the vector space, thus combining the advantages of global matrix factorization methods and local predictive methods.

Although modern language models like BERT or GPT have largely supplanted GloVe for many NLP tasks, it remains a fundamental tool for understanding word embeddings. Its pre-trained vectors (on Wikipedia, Common Crawl) are still used as a starting point in some applications, and its simplicity makes it an excellent educational tool for grasping the vector representation of language.