lecture 2

Question 1

areas of linguistics

Answer 1

1. phonetics
2. phonology
3. morphology
4. syntax
5. semantics
6. pragmatics

Question 2

phonetics

Answer 2

sounds of human language

Question 3

phonology

Answer 3

sound systems in human languages

Question 4

morphology

Answer 4

formation and internal structure of words

Question 5

syntax

Answer 5

formation and internal structure of sentences

Question 6

semantics

Answer 6

meaning of sentences

Question 7

pragmatics

Answer 7

study of the way sentences with their semantic meanings are used for particular communicative goals

Question 8

NLP and text

Answer 8

much of NLP focuses on text only, leaving out many layers of natural language

e.g., phonetics/phonology

Question 9

natural language is

Answer 9

1. compositional
2. arbitrary
3. creative
4. displaced

Question 10

compositional

Answer 10

the meaning of a sentence is the sum of the meaning of individual words (semantics) and how they are combined (syntax)

[set of rules that define grammaticality] + [lexicon of words that relate to the world we want to talk about]

meaning of an expression = semantics + syntax

Question 11

arbitrary

Answer 11

the link between form and meaning is arbitrary

Question 12

creative

Answer 12

every language can create an infinite number of possible new words and sentences

Question 13

displaced

Answer 13

we can talk about thing that are not immediately present

Question 14

human natural language

Answer 14

1. there is a critical period for acquiring language
2. children need to receive real input to acquire language
3. language is interconnected with other cognitive abilities

Question 15

structure & grammar

Answer 15

1. structure dictates how we can use language
2. we implicitly know complex rules about structure.
3. a community of speakers share a rough consent of their implicit rules. a grammar attempts to describe these rules.

Question 16

descriptive linguistics

Answer 16

how language is studied

focuses on describing how language is used in practice, without making judgments about correctness.

aims to objectively analyze and document rules that speakers naturally follow

Question 17

prescriptive linguistics

Answer 17

how language is taught

prescribes rules about how language should be used

often involves enforcing traditional rules and norms, which may not reflect actual usage

Question 18

language rules in education (grammar)

Answer 18

the rules taught as part of language education often serve purposes beyond describing the language

they often reflect social, cultural, and political influences

Question 19

grammaticality

Answer 19

a community of speakers share a rough consent of their implicit rules.

• all utterances we can generate from these rules are grammatical
• if we cannot produce an utterance using these rules, its ungrammatical

1. SVO order
2. subject & object pronouns
3. sentences can be grammatically correct without any meaning
4. idiolects

Question 20

grammaticality rules accept useless utterances and block out communicative utterances.

why do we need rules?

Answer 20

• if we ignore rules because we know what was probably intended, we actually limit possibilities
• rules give us expressivity

Question 21

NLP before self-supervised learning

Answer 21

the way to approach NLP was through understanding the human language system, and trying to imitate it (rule-based)

• probing
• reverse engineering

Question 22

probing

Answer 22

small unsupervised models that are trained to extract linguistic information from another model’s output

this helps understand how well different layers of an LLM capture various linguistic features

Question 23

reverse engineering language

Answer 23

1. syntax: parse the input to understand its grammatical structure
2. semantics: interpret meaning of the parsed input
3. discourse: understand the broader context and relationships between sentences

the process involves using theories to inform the design of NLP models, ensuring they can parse, understand, and generate human language

Question 24

testing an LLMs understanding of syntax

Answer 24

1. jabberwocky sentences
2. learning to apply grammatical rules from vast amounts of text data
3. word order
4. lexical generalization

Question 25

jabberwocky sentences

Answer 25

testing whether language models can **understand and represent syntactic structures**, even with nonsensical or novel words to see if a model's latent space encodes structural information

Question 26

word order

Answer 26

word order determines what a **subject** and what an **object** is a model's **latent space** can represent syntactic roles based on the positions of words in the space LLMs often dont care about word order, which affects its ability to grasp syntactic roles

Question 27

lexical generalization

Answer 27

task: semantic interpretation testing how models handle **novel words within familiar syntactic structures** --> i.e., whether models can generalize learned structures COGS measure generalization is hard for seq2seq models, not as hard for models with structure built in --> structure-aware models are therefore better

Question 28

COGS measure

Answer 28

tests models on their ability to generalize compositions of words and structures, by ensuring words/structures are different between the training and test set

Question 29

why is NLP hard

Answer 29

1. ambiguity 2. sparse data due to zipf's law 3. variation 4. expressivity 5. context dependence 6. unknown representation 7. spoken & grounded

Question 30

ambiguity

Answer 30

language has ambiguity at many levels --> word senses, POS, syntactic structure, quantifier scope, multiple meanings solution: 1. non-probabilistic methods: return all possible analyses 2. probabilistic models: return the best possible analyses --> 'best' is only good if our probabilities are accurate

Question 31

statistical NLP

Answer 31

1. typically more robust than earlier rule-based methods 2. relevant statistics/probabilities are learned from data 3. normally requires lots of data about any **particular phenomenon**

Question 32

sparse data due to zipf's law

Answer 32

- rank-frequency distribution is an inverse relation - f * r = k - a small number of words are very common, while the **majority of words is rare**, making it difficult for models to learn effectively from limited instances

Question 33

variation

Answer 33

- POS tags are trained on formal language, which makes it hard to use on informal language such as seen on social media - different contexts, vocabulary, grammatical structures, and varieties in language can reduce the tagger's effectiveness

Question 34

expressivity

Answer 34

- one form can have different meanings --> e.g., bank - the same meaning can have different forms --> e.g., 'some kids popped by' vs 'some children visited'

Question 35

context dependence + unknown representation

Answer 35

correct interpretation is context-dependent and often requires world knowledge --> e.g., interpretation of 'he dropped the ball'

Question 36

the role of meaning in linguistic structure

Answer 36

the meaning of words contributes to the overall structure and coherence of language

Question 37

how models are trained

Answer 37

1. input with rich semantic embedding 2. add positions to embeddings 3. masked self-attention 4. feed-forwards 5. linear transformations 6. softmax 7. probabilities

Question 38

differential object marking

Answer 38

different languages handle syntactic representation of objects within a sentence in different ways understanding differential object marking is crucial for NLP systems to accurately process and generate language representations across diverse languages LMs are aware of these gradations, but **animacy** influences this grammatical distinction --> animate entities are more likely to be subjects (agents)

Question 39

maybe not all structure-word combinations are possible

Answer 39

**making less plausible situations more explicit** is a common feature of grammatical structure

Question 40

meaning is not always compositional

Answer 40

1. idioms and metaphors: meaning cannot be directly inferred from the words themselves 2. we're constantly using constructions that we couldn't get from just a syntactic + semantic parse i.e., language **understanding requires more than just analyzing individual words** and their immediate syntax

Question 41

balancing surface-level memorization and deeper understanding within NLP models

Answer 41

**high dimensional spaces** are much better at capturing specific subtleties than any rules we could come up with