Test 6 Deep Learning

Question 1

2012

Answer 1

AlexNet - Alex Krizhevsky, Hinton, Sustlever - revolutionized the world of computer vision blowing other competition out of the water in the ImageNet competition

Question 2

Bag of Words - feature vector size

Answer 2

100,000 (vocabulary size)

Question 3

language model

Answer 3

probability distribution of a string

Question 4

what is the probability of a sequence of words?

Answer 4

product (big pi) of P(wm|w_1:m-1)

Question 5

What is the issue with computing the entire probability of a word given the previous sequence

Answer 5

for large sequences (large m) it is not possible

Question 6

n-gram

Answer 6

an n-gram is the probability of a word given n-1 previous words

Question 7

Maximum likelihood estimator for an n-gram

Answer 7

C(wm−[n−1]:m−1wm)
divided by
Sum C(wm−[n−1]:m−1w)

which is the same as
C(wm−[n−1]:m−1wm)
/C(wm−[n−1]:m−1)

Question 8

Perplexity

Answer 8

Weighted average branching factor -

Mth root of (1/P(w1:M)

Question 9

branching factor

Answer 9

number of possible next words that can follow an word

Question 10

Greedy Decoding

Answer 10

Wm = argmax_w P(w|w1:m-1) - choose the most likely next word

Question 11

Sampling Decoding

Answer 11

Cover the probability space with intervals equal to each word frequency - sample from a [0, 1] uniform distribution

Question 12

Top-K Sampling

Answer 12

Consider the K most likely entries

Question 13

Top-p sampling

Answer 13

consider the smallest set of entries whose cumulative probability exceeds p

Question 14

Beam Search

Answer 14

Generate K hypotheses
Extend each hypothesis by K (thus bringing total hypotheses to K^2)

Keep K of the K^2 hypotheses - rinse and repeat

Question 15

What is the distance between any two words in one-hot representation?

Answer 15

2 - all words are equidistant

Question 16

Distributional Hypothesis

Answer 16

Words that occur in similar contexts tend to have similar meanings

Question 17

Cosine Similarity Formula

Answer 17

uTv/||u||||v||

Question 18

Cosine Similarity Geometric Interpretation

Answer 18

The angle between the two normalized vectors

Question 19

Word Co-Occurence Matrix

Answer 19

Count how many times each word appears when looking at a similar word within a specific context window - size is vocab x vocab

Question 20

Word Vector

Answer 20

co-occurence matrix but for a single word

Question 21

What is the issue tracking cosine similarity for word vectors?

Answer 21

They’re pretty sparse - close to zero

Question 22

Word Embedding

Answer 22

C (co-occurence matrix) can be approximated with C = EU^T where E is of size v x d and U is of size p x d

Question 23

What is the rank of the co-occurence matrix vs the embedding matrices?

Answer 23

Full Rank vs Maximum rank D

Question 24

How can you calculate E and U^T?

Answer 24

Do singular value decomposition and consider the first d components

Question 25

How do you train a fully connected architecture for word embeddings?

Answer 25

Given a corpus, let V = {w|C(w) >= Cmin}, |V| = v Split the corpus into overlapping windows of size M Initialize a random embedding matrix Represent w1:M as [e1, e2,..., em] = so x^T is of size R 1xMd Use one-hot wM+1 as target

Question 26

Why is contextualized word representation necessary?

Answer 26

Words have multiple meanings, so we need representations that reflect the context, leading to BERT

Question 27

2018

Answer 27

BERT - bi-directional encoder representations from transformers

Question 28

What is the input and output of the RNN language model represented both as a function of the representation AND as a sequence

Answer 28

{x[m]} ^M-1_m=1 -> Emb(w1:m-1) {y[m]}^M-1m=1 = one-hot(w2:M)

Question 29

What are the RNN limitations for machine translation

Answer 29

Nearby context bias, fixed context size, slow sequential data processing

Question 30

2015

Answer 30

Bahdanau, Cho, Neural Machine Translation by Jointly Learning to Align and Translate - Attentional seq2seq model - dynamic focus on relevant parts of input sequence

Question 31

How is the hidden state of the decoder generated?

Answer 31

Project the hidden state at t-1 of the decoder on the hidden state of the encoder S^T h[n-1]

Question 32

Target Language Decoding

Answer 32

Autoregressive greedy decoding Get y~[1] after feeding Emb() to the decoder, feed Emb(argmax(y[1])) to the decoder and get y~[2]. Do until argmax(y~[n]) =

Question 33

What do you calculate for self-attention and how often?

Answer 33

For every x in {xm}M=1m=1 = X, compute query = Wqx key k = Wkx For the entire X, compute K = WkX query-key match r = K^T q a = softargmax(r) c = Va

Question 34

Describe the unique properties of c, the self-attention vector

Answer 34

it is permutationally-invariant to X, but permutationally equivariant to x

Question 35

How do you encode positional information in a set?

Answer 35

A sequence can be cast to an augmented set by adding input representation information about the index: additive positional encoding {x[m]}M, m=1 -> {xm + Enc(m))}M m=1

Question 36

2017

Answer 36

Vaswani et al - Attention is all you need replaced recurrence and convolutions with self-attention mechanisms for sequence modeling.