Block 4

Question 1

Why evaluate

Answer 1

Business reputation, Financial success, Being relevant in complex, real world, keeping up with tech

Question 2

Business reputation

Answer 2

Negative feedback can damage company

Question 3

Financial success

Answer 3

Appropriate evaluation carried out at right points in interaction process is a prudent investment

Question 4

Being relevant in complex, real world

Answer 4

By testing with real people in real contexts, you can pick up on cultural aspects that you might not have anticipated

Question 5

Keeping up with tech

Answer 5

Evaluating with users allows designers to experiment with prototypes of novel models of interaction, and gain fresh insights into emerging interaction paradigms

Question 6

Iterative design

Answer 6

Need at least 15 users to discover all usability problems in design

Better to distribute evaluations over smaller groups

If you have funding for 15 users spend this on 3 studies with 5 users in each

Question 7

Evaluation methods

Answer 7

User testing - Investigation of whether there is a need for the design

Usability testing - Evaluation of whether the system is usable by the intended users

Question 8

Choosing and combining methods

Answer 8

Many methods available in evaluation toolkit

Typically finding a fault requires a combination of methods

Not all evaluations need to be detailed

Opportunistic evaluation is done informally and gets quick feedback. This is done early in the design process

Question 9

Evaluating for accessibility

Answer 9

Some instances in which experts may be used to provide info that cannot be ascertained from users

E.g. visual impairment, aurally challenged

Question 10

New methods

Answer 10

New methods and variations on existing methods constantly being added to evaluation repertoire

Question 11

Ethical issues and informed consent

Answer 11

Becoming increasingly sensitive area, due to potential to gather and disseminate large amounts of personal data quickly over internet

Question 12

Adapting interviews and instructions

Answer 12

Need to consider whether any instructions given to users need to be adapted

Question 13

Adapting time allowances

Answer 13

Users with disabilities often require more time to complete an evaluation than users without disabilities

Question 14

Adapting physical arrangements

Answer 14

Need to consider whether the setting needs to be adapted

Question 15

When a helper, interpreter or advocate may be needed

Answer 15

Times when helper or user advocate needed to work alongside the participant

Question 16

Evaluating with children

Answer 16

Increasingly, children included in design and evaluation of interactive products, and each evaluation must be adapted to them

Question 17

Informed consent

Answer 17

About protecting rights of all parties involved in an evaluation by providing them with all the info they need to decide whether or not to participate

Need agreement of participant to take part in evaluation, need their permission to record evaluation, need their permission to use and store data

Question 18

UK data protection act

Answer 18

controls how personal info used by organisation, businesses or government

Everyone responsible for using data has to follow strict rues called ‘data protection principles’

Question 19

Any data given must -

Answer 19

Be used fairly and lawfully
used for limited, specifically stated purposes
used in a way that is adequate, relevant and not excessive
Accurate
Kept for no longer than absolutely necessary
handled according to people’s data protections rights
Kept safe and secure
Not transferred outside European economic area without adequate protection

Question 20

From data to info

Answer 20

Any evaluation will produce data of some kind

Whatever nature of data is, purpose of analysing and interpreting data is to transform it into info relevant and useful to design process

Question 21

Making sense of data

Answer 21

Analysis, interpretation, presentation

Question 22

Analysis

Answer 22

Follow three steps -

1. Collating the data - Gathering all data collected and organising it for processing
2. Analysing and summarising data - Extracting patterns or other observations from collated data. These patterns are first step of making sense of data
3. Reviewing the data - Accessing whether usability and user experience goals for interactive product have been met

Question 23

Interpretation

Answer 23

Data interpretation in evaluation is the process of actively considering what caused the problems that have been identified, and what to do about them

Question 24

Three steps to interpretation

Answer 24

1. Finding causes for usability problems that have been identified during analysis and rating seriousness of each
2. Prioritising issues
3. Proposing changes to design or making other recommendations to address these problems

Question 25

Presentation

Answer 25

Best way of presenting data needs to be decided Through simple presentations often sufficient in iterative design process, when evaluations inform design as it progresses, sometimes stronger evidence needed

Question 26

Working with Quantitative data

Answer 26

Three groups of methods for summarising data - 1. Tabulations, charts and rankings 2. Descriptive statistics 3. Inferential statistics

Question 27

Tabulations, charts and rankings

Answer 27

Provide visual representation of your data

Question 28

Descriptive statistics

Answer 28

Such as mean, median and mode that describe data you have obtained

Question 29

Inferential statistics

Answer 29

Tests of statistical significance that give the probability that a claim arising from your data can be applied to your user population as a whole

Question 30

Analysing and interpreting quantitative data

Answer 30

if you include descriptive statistics in your report, consider whether they are generalisable

Question 31

Representing quantitative data visually

Answer 31

Important to think about, and choose, most appropriate data display for your results, and for whom your presenting your data

Question 32

Working with qualitative data

Answer 32

Three simple forms of qualitative analysis 1. identifying recurring patterns or themes 2. Categorising data 3. Analysing critical incidents

Question 33

Repeatability

Answer 33

Test of rigorous qualitative analysis. Can patterns or categories that emerge be articulated clearly enough for independent evaluator to apply with same result

Question 34

Interpreting outcomes of data analysis

Answer 34

Outcome of data analysis should be list of usability problems found during evaluation

Question 35

Usability defect

Answer 35

``` Irritates/confuses user Makes system hard to learn/install/use Causes mental overload of user Causes poor user performance Violates design standards on guidelines Reduces trust in the system Tends to cause repeat errors Could make product hard to market ```

Question 36

Identifying causes of usability defects

Answer 36

Need to look in greater depth at various sources of evaluation data you have collected

Question 37

Prioritising issues

Answer 37

You can assign severity ratings Can prioritise the list by assigning severity ratings to each defect

Question 38

Proposing changes, making recommendations

Answer 38

Recommendations likely to contain several points - Successes to build upon Defects to fix Possible defects or successes not proven

Question 39

Presenting findings

Answer 39

Key is to provide useful, well-founded info concisely and accurately, in a form appropriate to audience

Question 40

Planning an evaluation involving users

Answer 40

``` Determine goals and questions Choose approach and methods Plan data collection Address practical issues Consider any ethical issues Plan how to analyse, interpret and present data Assemble materials needed to support evaluation Pilot studies ```

Question 41

Determine goals and questions

Answer 41

Before evaluation be clear why you are doing it. What questions need to be answered? What sort of info do you need to answer question?

Question 42

Choose approach and methods

Answer 42

Choose appropriate evaluation approaches and methods to answer specific questions identified

Question 43

Plan data collection

Answer 43

What data for you need to answer evaluation questions? Is it feasible to collect that data? Usability goals typically addressed by examining user behaviour, which can be captured qualitatively or quantitatively

Question 44

Address practical issues

Answer 44

Who, What , Where ,how (constraints)

Question 45

Users (who)

Answer 45

What is profile of intended user? How would you characterise them?

Question 46

Task (what)

Answer 46

What tasks, and how many, do you want to evaluate? Why have you chosen these tasks?

Question 47

Different task you could include

Answer 47

- Core tasks frequently performed by user - Tasks that have some new design features or functionality added - Critical tasks, even though they may not be frequently used - Task you feel has to be validated with users for greater clarity and understanding of the design team

Question 48

Setting (where)

Answer 48

Where will evaluation take place? How will you set up and lay out the session?

Question 49

Equipment (How)

Answer 49

Do you need any equipment?

Question 50

Constraints

Answer 50

Are there any practical constraints?

Question 51

Consider any ethical issues

Answer 51

What are ethical considerations of the evaluation? (Task, users, location, data collection)

Question 52

Plan how to analyse, interpret and present data

Answer 52

In order to collect data effectively, you must understand what you mean to do with it, and how you mean to analyse it Always plan how you will analyse and interpret data before you conduct and evaluation

Question 53

Assemble materials needed to support evaluation

Answer 53

- Evaluation script - Helps to have detailed script of everything you will say to participants - Introductory and background info for participant - Informed consent - A task description - Data collection forms - Post session interview plan or questionnaire - Analysis plan

Question 54

Pilot studies

Answer 54

Often appropriate to run pilot studies to ensure choices made are practicable Pilot study is a small-scale trial to check evaluation

Question 55

Evaluator/observer bias

Answer 55

Participants behaviour can be altered just by observer watching them

Question 56

Evaluator/observer bias

Answer 56

Participants behaviour can be altered just by observer watching them

Question 57

Methodology (bias)

Answer 57

Way evaluation designed and conducted can introduce bias Should counterbalance the order in which users do tasks

Question 58

Reporting and analysis

Answer 58

Analysis undertaken and results reported will be influenced to some extent by choice of evaluator

Question 59

Cognitive walkthrough

Answer 59

- Will user know what to do? - Will users see how to do it? - Will users understand from the feedback whether action was correct or not?