Evaluating Interactive Systems

Question 1

What is Formative Evaluation?

Answer 1

Used in the early stages of a project to compare, assess and refine design ideas. Involves open research questions so that the researcher can learn more information to inform the design

Question 2

What is Summative Evaluation?

Answer 2

Used later in the stages of a project. Involves closed research questions to test and evaluate systems according to predefined criteria

Question 3

What is Analytical Evaluation?

Answer 3

Based on applying a theory to analyse and discuss the design. Analysing your design

Question 4

What is Empirical Evaluation?

Answer 4

Making observations and measurement of users. Collecting data for analysis

Question 5

What is Quantitative Data?

Answer 5

Numbers

Question 6

What is Qualitative Data?

Answer 6

Words, pictures, audio or video

Question 7

Are analytical methods used for formative or summative evaluation?

Answer 7

Formative

Question 8

List the qualitative analytical evaluation methods

Answer 8

1. Cognitive walkthrough
2. Cognitive dimensions of notations

Question 9

What is cognitive walkthrough useful for evaluating?

Answer 9

Closed research questions

Question 10

What is cognitive dimensions of notations useful for evaluating?

Answer 10

Open research questions

Question 11

What is Keystroke Level Model useful for evaluating?

Answer 11

To create numerical comparisons of closed research questions

Question 12

List the quantitative analytical evaluation methods

Answer 12

1. Keystroke Level Model

Question 13

List the quantitative empirical evaluation methods

Answer 13

1. A/B experiments
2. Controlled laboratory trials

Question 14

List the qualitative empirical evaluation methods

Answer 14

1. Think-aloud / ethnography
2. Interviews
3. Field observation
4. Surveys

Question 15

Are qualitative empirical methods used for formative or summative evaluation?

Answer 15

Formative

Question 16

Are quantitative empirical methods used for formative or summative evaluation?

Answer 16

Summative

Question 17

What 3 things do you need to run a Randomised Control Trial?

Answer 17

1. A performance measure
2. A representative sample of your target population
3. An experimental task that can be used to collect performance data

Question 18

What is Internal Validity? What factors does it include?

Answer 18

Asks “was the study done right?”
Includes factors: reproducibility, scientific integrity, refutability

Question 19

What is External Validity? What factors does it include?

Answer 19

Asks “does the study tell us useful things?” and focuses on if results can be generalisable to real world situations
Includes factors: representativeness of sample population, experimental task, application context

Question 20

How are the results of a randomised control trial measured?

Answer 20

In terms of effect size, possibly including correlation with factors that might affect performance

Question 21

What is reported as the results of a randomised control trial?

Answer 21

Significance measures are reported to check whether the observed effects might have resulted from random variation or other factors rather than the treatment

Question 22

Give 2 disadvantages of RCTs

Answer 22

1. Overcoming natural variation needs large samples
2. They do not naturally provide understanding of why a change occurred so it is hard to know if the effect will generalise. If there are many relevant variables that are orthogonal, many separate experiments might be required to distinguish between their effects and interactions

Question 23

What do companies tend to use instead of RCTs?

Answer 23

Proxy measures such as the number of days that customers continue actively using the product

Question 24

What must all controlled experiments be assessed according to?

Answer 24

Their internal and external validity

Question 25

Why is qualitative data often recorded and transcribed?

Answer 25

So it can be analysed using a reproducible scientific method

Question 26

Which qualitative data analysis method is used to answer closed questions? Give an example of a closed question

Answer 26

Categorical coding Eg. comparing different groups of people or users of different products

Question 27

Which qualitative data analysis method is used to answer open questions? What is an open question?

Answer 27

Grounded theory Used when there is no prior expectation of the insights the researcher is looking for

Question 28

What are the steps of categorical coding?

Answer 28

1. Create a coding frame of expected categories of interest 2. Text data is segmented (eg. on phrase boundaries) 3. Each segment is assigned to one category so that frequency and correspondence can be compared

Question 29

What is inter-rater reliability?

Answer 29

Two or more people make the coding decisions independently to avoid systematic bias or misinterpretation. They then compare how many decisions agree relative to chance using a statistical measure such as Cohen's Kappa (2 people) or Fleiss' Kappa (more). It may involve refining the coding frame to resolve decision criteria if there are still significant disagreements. This is used to 'prototype' the coding frame before proceeding to the main corpus

Question 30

Which qualitative data analysis method should incorporate inter-rater reliability?

Answer 30

Categorical coding

Question 31

What are the steps of grounded theory?

Answer 31

1. Open coding - read the data closely, looking for interesting categories 2. Collect fragments, write memos to capture insights as they occur 3. Emerging themes are organised using axial coding across different sources of evidence 4. Memos, themes and findings are constantly compared to the original data so they can be objectively justified Ends when the theoretical description has reached saturation in relation to the original data