Contextual embeddings and Transformers

Embeddings fixed a big problem: text could finally be represented as meaning-like vectors instead of just word counts.

But early embeddings had a quiet flaw.

They treated a word like it had one stable meaning everywhere.

"The bank approved my loan."
"The boat stopped near the river bank."

Same word. Different job.

That is where contextual embeddings start to matter.

This sits after NLP, NLU, NLG, and core language tasks, then counting words to embeddings.

Static embeddings

A static word embedding gives one vector to one word.

bank -> [0.12, -0.44, 0.91, ...]

That vector does not care whether the sentence is about money or a river.

"I deposited money in the bank."
"I sat near the river bank."

Static embeddings were still a massive upgrade over keyword counts.

They could learn that:

dog ~= cat
car ~= vehicle
happy ~= joyful

The problem is that they mostly answer:

What does this word usually mean?

Sometimes the better question is:

What does this word mean here?

Contextual embeddings

A contextual embedding changes depending on the surrounding sentence.

bank + loan  -> finance, account, credit
bank + river -> water, land, shore

That small shift is a big deal.

The model is no longer carrying around a dictionary-style vector. It is building a representation from the current sentence.

static embedding      = general word meaning
contextual embedding  = word meaning inside this context

This is why models like BERT, RoBERTa, T5, GPT-style models, and modern SentenceTransformers became so useful. They do not just embed words. They embed words after looking at the words around them.

Where embeddings sit

The usual pipeline is still text into numbers.

raw text
-> tokenization
-> token IDs
-> embeddings
-> model layers
-> output

The embedding layer starts the process:

"I"     -> [0.12, -0.44, 0.08, ...]
"love"  -> [0.91, 0.33, -0.10, ...]
"movie" -> [0.62, 0.18, -0.31, ...]

Then the model updates those representations.

That distinction matters:

embedding lookup      = initial token vector
model processing      = contextual representation

In modern systems, embeddings show up in two places:

inside models  -> token representations
outside models -> search, retrieval, clustering, similarity

Same word, slightly different use.

Self-attention

To understand bank in this sentence:

"The bank approved my loan."

the model should care about:

approved
loan

To understand bank here:

"The boat stopped near the river bank."

it should care about:

boat
river
near

Self-attention is the mechanism that lets tokens look at other tokens and decide what matters.

The core formula:

Plain version:

queries ask what a token is looking for
keys describe what each token offers
values carry the information to combine

The model compares queries with keys, gets attention weights, then mixes the value vectors using those weights.

The result:

initial token embedding
-> self-attention
-> contextual embedding

In encoder-style models, a token can usually attend to tokens on both sides. In decoder-only language models, a token usually attends to previous tokens, because future tokens have not been generated yet.

Position

Transformers also need order.

These two sentences contain the same words:

"Dog bites man."
"Man bites dog."

Not the same meaning.

So each token representation needs at least:

what token is this?
where is it?

The simple version:

Plain version: at position i, combine the token embedding e_i with position information p_i.

Different Transformer variants handle position differently, but the mental model holds:

token meaning + position signal

Without position, the model has a bag of tokens. With position, it can model sequence.

Transformer flow

A simplified Transformer pass looks like this:

text
-> tokens
-> token embeddings + position information
-> self-attention layers
-> contextual representations
-> output

That is enough to explain why Transformers became useful across translation, summarization, question answering, chat, code generation, semantic search, and document understanding.

They are good at repeatedly asking:

Which other tokens matter for understanding this token right now?

The evolution

The story connects cleanly:

one-hot encoding
-> Bag of Words
-> TF-IDF
-> static word embeddings
-> contextual embeddings
-> Transformers and LLMs

Each step fixed a real weakness.

one-hot      = identity
Bag of Words = counts
TF-IDF       = keyword importance
static emb   = general meaning
context emb  = meaning in this sentence
Transformers = relationships between tokens

The short version:

Traditional NLP counted words.
Embeddings represented meaning.
Transformers made meaning depend on context.