From counting words to embeddings

Before embeddings, a lot of NLP was basically counting.

Not in a bad way. Counting got surprisingly far.

The old pattern:

text
-> split into words
-> count words
-> feed counts into a model

A classifier could learn that love often shows up in positive reviews and hate often shows up in negative ones.

The missing piece was meaning.

car ~= automobile
cheap ~= affordable
movie ~= film

Counting words does not know that.

The path from word identity to meaning-like vectors starts here. Previous stop: NLP, NLU, NLG, and core language tasks. Next stop: contextual embeddings and Transformers.

One-hot encoding

One-hot encoding is the blunt version.

Vocabulary:

["cat", "dog", "car", "happy", "sad"]

Each word gets one active position:

cat   -> [1, 0, 0, 0, 0]
dog   -> [0, 1, 0, 0, 0]
car   -> [0, 0, 1, 0, 0]
happy -> [0, 0, 0, 1, 0]
sad   -> [0, 0, 0, 0, 1]

This captures identity, not meaning.

To the model:

cat and dog = different
cat and car = different

No built-in sense that cat and dog are closer than cat and car.

One-hot vectors identify words, but do not explain them

visual model

catdogcarhappysad

cat10000

dog01000

car00100

happy00010

sad00001

Each word owns one active position. Related words are still separate columns.

Math version:

v_j = \begin{cases} 1, & j = i \\ 0, & j \ne i \end{cases}

Plain version: turn on one position for the selected word and leave everything else at zero.

Bag of Words

One-hot encoding handles individual words.

Bag of Words handles a sentence or document by counting which words appear.

Vocabulary:

["I", "love", "hate", "this", "movie"]

Sentence:

"I love this movie"

Vector:

[1, 1, 0, 1, 1]

Another sentence:

"I hate this movie"

Vector:

[1, 0, 1, 1, 1]

For simple sentiment tasks, this can work. love pushes one way, hate pushes the other.

The weakness is order.

"Dog bites man."
"Man bites dog."

Bag of Words sees almost the same counts. Meaning is clearly not the same.

Bag of Words keeps counts and loses order

visual model

Ilovehatethismovie

I love this movie11011

I hate this movie10111

The model can see that love or hate appears, but it does not preserve sentence structure.

Math version:

x_j = \operatorname{count}(w_j, d)

Plain version: each vocabulary position stores how many times that word appears in the document.

TF-IDF

Raw frequency has another issue: common words show up everywhere.

the
is
and
of

They appear a lot, but usually do not say much about the document.

TF-IDF adds a useful correction:

important word = common in this document, uncommon across all documents

In an AI article:

transformer -> high signal
the         -> low signal

TF-IDF became useful for search, classification, recommendation, keyword extraction, and old-school retrieval.

Still, it is mostly keyword-based.

Query:
"cheap laptop"
 
Document:
"affordable notebook computer"

Humans see the match. TF-IDF may not.

TF-IDF lowers common words and raises specific words

visual model

the

and

transformer

attention

A word is useful when it is frequent in this document but not common across every document.

Math version:

\operatorname{tfidf}(t, d, D) = \operatorname{tf}(t, d) \cdot \operatorname{idf}(t, D)

\operatorname{idf}(t, D) = \log \frac{N}{\operatorname{df}(t)}

Plain version: a term scores higher when it appears often in this document but not in every document.

Many libraries smooth the formula:

\log \frac{1 + N}{1 + \operatorname{df}(t)} + 1

The semantic gap

The old methods share the same problem:

word identity != word meaning
word frequency != word meaning

Example:

car
automobile
vehicle

One-hot or Bag of Words treats these as separate columns.

car        -> [1, 0, 0]
automobile -> [0, 1, 0]
vehicle    -> [0, 0, 1]

No built-in closeness.

There is also the size problem. A vocabulary with 100,000 words creates giant sparse vectors.

cat -> [0, 0, 0, 0, 1, 0, 0, ...]

Most values are zero. Not much meaning, lots of space.

Embeddings

An embedding represents text as a dense vector of numbers.

The text can be:

a word
a token
a sentence
a paragraph
a document

The goal is simple: similar meanings should land close together.

king ~= queen
dog ~= cat
movie ~= film
happy ~= joyful
car ~= automobile

This is the shift from identity/frequency to meaning-like similarity.

Embeddings put similar meanings nearby

visual model

carvehicleautomobiledogcatfilmmovie

The axes are not literal. The useful idea is distance: close points mean related meanings.

Cosine similarity is one common way to compare vectors:

\operatorname{cosine}(a, b) = \frac{a \cdot b}{\lVert a \rVert \lVert b \rVert}

Plain version: compare vector directions. Similar direction, similar meaning.

Sparse vs dense

Old representations are often sparse.

cat -> [0, 0, 0, 0, 1, 0, 0, ...]

Embeddings are dense.

cat -> [0.23, -0.71, 0.44, 0.09, ...]

Many dimensions carry signal.

Math version:

\mathbf{e}_i = E[\operatorname{token\_id}_i]

E \in \mathbb{R}^{|V| \times d}

Plain version: use the token ID to look up a row in an embedding table.

The shift:

sparse vectors -> dense vectors
word identity  -> meaning-like similarity
exact matching -> semantic search

Static word embeddings

Static word embeddings give one vector per word.

bank -> [0.12, -0.44, 0.91, ...]

The problem: the vector does not change.

"I deposited money in the bank."
"I sat near the river bank."

Same bank vector, different meaning.

Static embeddings were still a big upgrade over Bag of Words because related words could land close together:

cat close to dog
car close to vehicle
happy close to joyful

But they mostly capture general meaning, not sentence-specific meaning.

Word embeddings vs text embeddings

Two separate questions matter.

First:

What is being embedded?

Could be:

word
token
sentence
paragraph
document

Second:

Does the vector change with context?

Could be:

static
contextual

Text embeddings are heavily used in semantic search and RAG.

Typical flow:

document
-> split into chunks
-> embed each chunk
-> store vectors

Query flow:

question
-> embed question
-> compare with document vectors
-> retrieve closest chunks

That is why:

"How do I change my password?"

can match:

"Users can reset their password by clicking Forgot Password on the login page."

Different words. Close meaning.

The short version

The path so far:

one-hot encoding
-> Bag of Words
-> TF-IDF
-> static word embeddings
-> text embeddings

Each step fixes something.

one-hot = word identity
Bag of Words = word counts
TF-IDF = word importance
embeddings = meaning-like similarity

The next unresolved problem:

same word, different sentence, different meaning

That is where contextual embeddings and Transformers enter.