C5

Question 1

How to get example data?

Answer 1

1. use existing labelled data
2. create new labelled data

Question 2

3 types of existing labelled data

Answer 2

1. benchmark data
2. existing human labels
3. labelled user-generated content

Question 3

benchmark data and pros and cons

Answer 3

used to evaluate and compare methods, often created in the context of shared tasks

advantages:
- high-quality
- re-usable
- compare results to others

disadvantages:
- not availale for every specific problem and data type

Question 4

existing human labels with pros and cons

Answer 4

labels that were added to items by humans, but not originally created for training machine learning models, eg.
the international patent classification system (millions of patents manually classified in a hierarchical classification system by patent experts)

advantages:
- high-quality
- potenially large
- often freely available

disadvantages:
- not available for every specific problem and data type
- not always directly suitable for training classifiers

Question 5

labelled user generated content with pros and cons

Answer 5

• hashtags on Twitter (eg. learning sarcasm #not)
• scores and aspects in customer reviews (to learn sentiment and opinion)
• likes of posts to learn which comments are the most interesting

advantages:
- potentially large
- human-created
- freely available, depending on platform

disadvantages:
- noisy: often inconsistent
- may be low-quality
- indirect signal

Question 6

5 steps to create new labelled data

Answer 6

1. create a sample of items
2. define a set of categories
3. write annotation guidelines version 1
4. test and revise the guidelines with new annotators until the guidelines are sufficiently clear
   - task should be clearly defined, but not trivial
   5 human annotation (experts or crowdsourcing)
5. compare the labels by different annotators to estimate the reliability of the data (inter-rater agreement)

Question 7

crowdsourcing

Answer 7

useful for tasks that humans are typically good at while computers need a lot of examples to do it properly and where no experts are needed

main challenge: quality control =>
- don’t pay too little
- have a check in the task set-up
- say that their work is compared to expert annotations
- measure inter-rater agreement

Question 8

why would we compute the inter-rater agreement?

Answer 8

• human labelled example data is the truth for the classifier
• but 2 human classifiers never fully agree
• so we always have part of the example data labelled by 2 or 3 raters and compute the inter-rater agreement => know the reliability of the example data and measure the difficulty of the task
• Cohen’s Kappa

Question 9

Cohen’s Kappa

Answer 9

k = (Pr(a) - Pr(e)) / (1 - Pr(e)) (example on slide 54)

Pr(e) = Pr(e, c1) + Pr(e,c2) etc.