Interactive embeddings

Contextual vs Static Embeddings

Change the sentence and watch one token keep the same static vector while its contextual vector changes through Transformer attention.

Selected stage

Self-attention

Each token compares query to keys, gets weights, and mixes value vectors from relevant tokens.

bank attends to loan in one sentence and river in another.

Sentence

Thebankapprovedmyloan

The token bank is pulled toward finance because approved and loan dominate the context.

Attention weights for "bank"

The4%

bank28%

approved25%

my5%

loan38%

Static vector

0.400.12-0.220.71-0.18

Contextual vector

0.780.55-0.080.320.64

Static similarity for bank1.00

The static vectors are identical, so finance-bank and river-bank look the same at lookup time.

Contextual similarity for bank0.03

After attention, the contextual vectors separate because each sentence supplies different evidence.

Contextual embeddings mean instances

Static embeddings represent a word type. Contextual embeddings represent each occurrence of that word inside a specific sentence.

bank at position 2

The bank near the river was muddy

river-side place

bank at position 9

The river bank was owned by a bank

financial institution