L16 - Word Embedding

Question 1

What issue does Word Embedding solve?

Answer 1

Topic Modellings inability to work effectively on shorter texts.

Question 2

Give a high level explanation of the concept of Word Embedding…

Answer 2

Converts words into numeric vectors. Similar words can then be represented by the same numeric value within the Word Embedding.

Question 3

Give the step by step process of how it works…

Answer 3

1. Apply one hot vector transformation to all words in the corpus.
2. Input vectors into a 2D neural-network. Input and output vectors will alwars represent 1 word has a complexity of 2V + N (V is input vecotrs, N is hidden layer)
3. Input vectors are fed to activation function along with their associated weight.
4. Each input vector now corresponds with a value in Word Embedding (middle vector). Similar words are given similar or same value. Thus, next word can be predicted.

Question 4

What are 2 adaptations to improve the Word Embedding model?

Answer 4

Continuous Bag of Words Model - As opposed to a single vector for each input, use a set of vectors (set of words) to establish context for a more accurate output.

Skip-gram model - Enables the prediction of more than 1 word.

Question 5

What is the goal of Word Embedding?

Answer 5

We want to establish a low dimension Word Embedding (weight vector) which enables the prediction of next words.

Question 6

What are the 2 main properties of Word Embeddings?

Answer 6

• Numerical Vectors - Represent words as numerical vectors.
• Vector Space Relationships - In the mathematical vector space, closeness of vectors represent the similarities of the words the vectors represent.

Question 7

What is the main issue of Word Embedding? What is the solution to this?

Answer 7

Polysemy - Some words can be viable for multiple embeddings due to multiple meanings depending on the context. For example, Queen could be royalty, music and cards. If the text contains all these themes, no clear relationship can be established between the word and document.

Solution : Sense Embedding

Question 8

How does Sense Embedding solve the Polysemy issue of Word Embedding?

Answer 8

Construct embeddings for each sense that the word can be used in.

Feed the model with many labeled examples in which the word is used.
The model can then derive more detailed context of when the word is used and it’s associated data via the appropriate embedding.

Question 9

What does sentence / document embedding mean?

Answer 9

The closer proximity of sentence or document vectors implies that they are more similar.

Question 10

What are some applications of Word Embedding?

Answer 10

Automatic translation
Text summarisation
Clustering similar texts