Lec 6 | Language

Question 1

It spans all tasks where the AI gets human language as input.

Answer 1

Natural Language Processing (NLP)

Question 2

the AI is given text as input and it produces a summary of the text as output.

Answer 2

automatic summarization

Question 3

the AI is given a corpus of text and the AI extracts data as output

Answer 3

information extraction

Question 4

the AI is given text and returns the language of the text as output.

Answer 4

language identification

Question 5

the AI is given a text in the origin language and it outputs the translation in the target language.

Answer 5

machine translation

Question 6

the AI is given text and it extracts the names of the entities in the text (for example, names of companies).

Answer 6

named entity recognition

Question 7

the AI is given speech and it produces the same words in text.

Answer 7

speech recognition

Question 8

the AI is given text and it needs to classify it as some type of text.

Answer 8

text classification

Question 9

where the AI needs to choose the right meaning of a word that has multiple meanings (e.g. bank means both a financial institution and the ground on the sides of a river).

Answer 9

Word Sense Ambiguation

Question 10

A sentence structure

Answer 10

Syntax

Question 11

The meaning of words or sentences

Answer 11

Semantics

Question 12

A system of rules for generating sentences in a language.

Answer 12

Formal Grammar

Question 13

The text is abstracted from its meaning to represent the structure of the sentence using formal grammar.

Answer 13

Context Free Grammar

Question 14

What do these non-terminal symbols mean?

• N
• V
• NP
• VP
• S
• D
• P
• ADJ

Answer 14

• Noun
• Verb
• Noun Phrase
• Verb Phrase
• Sentence
• Determiner
• Preposition
• Adjective

Question 15

A sequence of n items from a sample of text.

Answer 15

n-gram

Question 16

What n-gram is this?

the items are characters

Answer 16

character n-gram

Question 17

What n-gram is this?

the items are words

Answer 17

word n-gram

Question 18

Continous sequence of items from a sample of text. They may have sequences of 1, 2, or 3.

3 answers

Answer 18

unigram, bigram, and trigram

Question 19

Where can we use/implement n-gram?

Answer 19

It is useful for text-processing.

Since some words occur together more often then others, it is possible to also predict the next word with some probability. A helpful step in natural language processing is breaking the sentence into n-grams.

Question 20

the task of splitting a sequence of characters into pieces (tokens).

Answer 20

Tokenization

Question 21

Tokens can be words as well as sentences, in which case the tasks are called what?

Answer 21

word tokenization or sentence tokenization

Question 22

What are challenges faced when using splitting words? How do we deal with them?

Answer 22

Words with apostrophes (e.g. “o’clock”) and hyphens (e.g. “pearl-grey). Additionally, some punctuation is important for sentence structure, like periods. Dealing with these questions is the process of tokenization.

Question 23

• Consists of nodes, the value of each of which has a probability distribution based on a finite number of previous nodes.
• Can be used to generate text.

Answer 23

Markov Models

Question 24

How do we use Markov Models?

Answer 24

we train the model on a text, and then establish probabilities for every n-th token in an n-gram based on the n words preceding it.

Question 25

* A model that represents text as an unordered collection of words. * This model ignores syntax and considers only the meanings of the words in the sentence. * This approach is helpful in some classification tasks, such as sentiment analysis.

Answer 25

Bag-of-Words Model

Question 26

This can be used, for instance, in product reviews, categorizing reviews as positive or negative.

Answer 26

Sentiment Analysis

Question 27

a technique that’s can be used in sentiment analysis with the bag-of-words model.

Answer 27

Naive Bayes

Question 28

What problem might we run into when using Naive Bayes? And how do we solve this problem?

Answer 28

One problem that we can run into is that some words may never appear in a certain type of sentence. One way to go about this problem is with Additive Smoothing.

Question 29

It is where we add a value α to each value in our distribution to smooth the data. This way, even if a certain value is 0, by adding α to it we won’t be multiplying the whole probability for a positive or negative sentence by 0.

Answer 29

Additive Smoothing

Question 30

A specific type of additive smoothing where it adds 1 to each value in our distribution, pretending that all values have been observed at least once.

Answer 30

Laplace Smoothing

Question 31

the task of finding relevant documents in response to a user query.

Answer 31

Information Retrieval

Question 32

* We use this ti achieve Information Retrieval. * Models for discovering the topics for a set of documents.

Answer 32

Topic Modeling

Question 33

How can the AI go about extracting the topics of documents? And how does it work?

Answer 33

One way to do so is by looking at term frequency, which is simply counting how many times a term appears in a document.

Question 34

words that are used for syntactic purposes in the sentence and not as independent units of meaning, such as "am", "do", "is", "with", “the,” “by,” “and,” “which,” “yet,” etc.

Answer 34

Function Words

Question 35

words that carry meaning independently, such as “crime,” “brothers,” “demons,” “gentle,” “meek,” etc.

Answer 35

content words

Question 36

which is a measure of how common or rare a word is across documents in a corpus.

Answer 36

Inverse Document Frequency

Question 37

# GIVE EQUATION Inverse Document Frequency

Answer 37

``` log(TotalDocuments / NumDocumentsContaining(word) ```

Question 38

* A library that can rank what words are important in a document. * This library is capable of multiplying the term frequency of each word by the inverse document frequency, thus getting a value for each word.

Answer 38

Term frequency - inverse document frequency (tf-idf)

Question 39

The task of extracting knowledge from documents. This task can take the form of giving a document to the AI as input and getting a list of companies and the years when they were founded as output.

Answer 39

Information Extraction

Question 40

A database similar to a dictionary, where words are given definitions as well as broader categories.

Answer 40

Word Net

Question 41

each word is represented with a vector that consists of as many values as we have words. Except for a single value in the vector that is equal to 1, all other values are equal to 0.

Answer 41

One-hot Representation

Question 42

Give a problem or drawback of one-hot representation. What is the solution to this problem/drawback?

Answer 42

* It will be incredibly inefficient when representating more words (if you have 50000 words there will be 50000 vectors with the length of 50000 each). * Another problem in this kind of representation is that we are unable to represent similarity between words like “wrote” and “authored.” Solution: we turn to the idea of Distributed Representation

Question 43

Meaning is distributed across multiple values in a vector. Each vector has a limited number of values. It allows us to generate unique values for each word while using smaller vectors. We are able to represent similarity between words by how different the values in their vectors are.

Answer 43

Distributed Representation

Question 44

an algorithm for generating distributed representations of words

Answer 44

word2vec

Question 45

# CS50 QUIZ Consider the below context-free grammar, where S is the start symbol. S -> NP V NP -> N | A NP A -> "small" | "white" N -> "cats" | "trees" V -> "climb" | "run" Consider also the following four sentences. Cats run. Cats climb trees. Small cats run. Small white cats climb. Of the four sentences above, which sentences can be derived from the context-free grammar?

Answer 45

Sentence 1, Sentence 3, and Sentence 4.

Question 46

# CS50 QUIZ Which of the following is not a true statement? * Attention mechanisms can be used to determine which parts of an input sequence are most important to focus on. * One-hot representations of words better represent word meaning than distributed representations of words. * Transformers can be faster to train than recurrent neural networks because they are more easily parallelized. * A Naive Bayes Classifier assumes that the order of words doesn’t matter when determining how they should be classified.

Answer 46

One-hot representations of words better represent word meaning than distributed representations of words.

Question 47

# CS50 QUIZ Why is “smoothing” useful when applying Naive Bayes? * Smoothing allows Naive Bayes to better handle cases where evidence has never appeared for a particular category. * Smoothing allows Naive Bayes to better handle cases where there are many categories to classify between, instead of just two. * Smoothing allows Naive Bayes to be less “naive” by not assuming that evidence is conditionally independent. * Smoothing allows Naive Bayes to turn a conditional probability of evidence given a category into a probability of a category given evidence.

Answer 47

Smoothing allows Naive Bayes to better handle cases where evidence has never appeared for a particular category.

Question 48

# CS50 QUIZ From the phrase “must be the truth”, how many word n-grams of length 2 can be extracted?

Answer 48

3