Observation Flashcards

Question

Think aloud preperation

Answer 1

* explain the system * explain the setting * explain expectation 1. using the scenarios prepared earlier, write a draft list of tasks 2. try out the tasks and estimate how long they will take a participants to complete 3. prepare a task sheet for the participants 4. get ready for the test session 5. tell the participants that it is the system that is under test, not them; explain and introduce tasks 6. participants start the tasks. Have them give you running commentary on what they are doing, why they are doing it and difficulties or uncertainties they encounter 7. encourage participants to keep talking 8. When the participants have finished, interview them briefly about the usability of the prototype and the session itself. Thank them 9. write up your notes as soon as possible and incorporate into a usability report

Answer 2

* qualitative, subjective mostly * ethnographic, delivers to the point experience for specific issues / problems * generalisations are very difficult, require high level of experience * interpretations can be done based on various different psychological theories and models

Answer 3

* People's use of technology in their everyday lives can be evaluated in living labs * such evaluations are too difficult to do in a usability lab

Answer 4

* Are field studies, not lab studies * In situ, means result includes measurements of the context * context and situation is not controlled * such studies are more expensive * more likely to find novel insight and experience * Ubicomp studies requrie additional effort * Ubicomp studies e.g. normally also require control conditions, prestudies, calculation of number of participants, selection of participants, data selection and statistics

Answer 5

study current behavior: * what are people doing now proof-of-concept studies: * does my technology function in the real world experience studies. * how does using my prototype change people's behaviour or allow them to do new things

Answer 6

good for proof of concept person simulates and controls system from behind the scenes * use mock interface and interact with users * good for simulating system that would be difficult to build

Answer 7

Surveys * often used as prestudy * carried out after any change of condition in a between-subject study * regular in-between survey while a study to measure change of participants reaction Logging * use the mobile device to also collect data about usage

Answer 8

how will you use the logged data? * select appropriate data to log (at the right frequency) make a list of specific questions that you expect to answer from the log data will your logging help you know if the study is going smoothly?

Answer 9

A system of tools and techniques for optimizing web usage by measuring, collecting, analyzing and reporting web data typically focus on the number of web visitors and page views.

Answer 10

ESM is a study method using questionnaires Participants are asked to fill out short questionnaires at various points throughout the day You get a different picture than to recall later Considerations: * how often to ask the participant * how many questions * collect experience or sensor information

Answer 11

For any study: A: start with a concrete research question B: answer the following questions: * what will your participants do during the study * what data will you collect * how long will the study be

Answer 12

1. Have a clear research goal and question 2. Create a study design document containing * 1. Research question / Hypothesis * 2. Detailed participant Profile * 3. Detailed method description (what will part. do) * 4. Detailed timeline description * 5. Types of Data you collect * 6. Analysismethod * 7. How you draw conclusion / validate hypothesis

Answer 13

Depends on type of study * experience studies (several weeks) are longer than proof of concept studies (serveral days) * studies of current behaviour may start from hours to weeks Depends on novelty * usage of novel systems is often very different at the start (enthusiasm or scepticism) and after longer period of use Practical considerations * If it requires much effort from the participants you have to restrict measurement time * Frequency of need for interaction with participants: High frequency means shorter measurement time Frequency of use * High frequency of use reduces measurement of time required

Answer 14

*Reliability does the method produce the same results on separate occasions? * Validity does the method measure what it is intended to measure internal validity - external validity * Ecological validity does the environment of the evaluation distort the results? Is the result transferable to a general environment? * Biases: Are there biases that distort the results? * Scope: How generalizable are the results

Answer 15

First you have to answer 3 questions before you start: * representation of participants to the intended user group * grouping of participants * data sampling strategy

Answer 16

* Representative Participant Set * Non-Representative Participation Set * Be careful: Many statistics assume a representative set

Answer 17

one group only or multiple groups group selection based on * self-reported experience * frequency of use * amount of experience * demographics * different activities the participants have to perform

Answer 18

Random sampling * everyone has equal probability of being selected as participant based on a list Systematic Sampling * Based on predefined criteria, e.g. every 10th person entering the ECE Center Stratified sampling * Additionally, it is important to select people reflecting the distribution in your intended user group. So you care e.g. that your final set contains 50% male and 50% female Samples of convenience * Volunteer based. Must be adjusted to the wanted user group

Answer 19

Depends on acceptable error! * Major problems can be identified by 3-4 people. * Early stage design require less participants * But this is an oversimplification --> Perform a pre-test where participants have to first detect known usability issues, caluculate averatge percentage of found usability issues over all participants Gives you the percentage of found issue in average

Answer 20

Participants learn fast - test order may have a significant influence on the outcome of the experiment --> Reschedule order of tasks for each participant This is not necessary with unrelated tasks. Sometimes it is impossible because tasks depend on each other

Answer 21

* Experts use their knowledge of users & technology to review software usability * Expert critiques can be formal or informal * Heuristic evaluation is a review guided by a set of heuristics * Walkthroughs involve stepping through a pre-planned scenario noting potential problems.

Answer 22

* Visibility of system status * Match between system and real world * user control and freedom * consistency and standards * error prevention * recognition rather than recall * flexibility and efficiency of use * aesthetic and minimalist design * help users recognize, diagnose, recover from errors * hep and documentation

Answer 23

1. briefing session to tell experts what to do 2. Evaluation period of 1-2 hours in which * each expert works separately * take one pass to get a feel for the product * take a second pass to focus on specific features 3. Debriefing session in which experts work together to prioritize problems.

Answer 24

+ few ethical problems - no users involved + few practical problems - no users involved - can be difficult to find experts - important problems may get missed - many trivial problems are often identified - experts have biases

Answer 25

* focus on ease of learning and or usage * designer presents an aspect of the design & usage scenarios * Expert is told the assumptions about user population, context of use, task details * one or more experts walk through the design prototype with the scenario * experts are guided by questions

Answer 26

1. Will the correct action be sufficiently evident to the user? 2. Will the user notice that the correct action is available? 3. Will the user associate and interpret the response from the action correctly? 4. If correct action is performed, will the user see that progress is made towards his goals?

Answer 27

variation on the cognitive walkthrough, performed by a carefully managed team The panel of experts begins by working separately Then there is managed discussion that leads to agreed decisions The approach lends itself well to participatory design

Answer 28

Sensitivity * index must be sensitive to changes in task difficulty or resource demand Selectivity * index should NOT be sensitive to changes unrelated to resource demands Diagnosticity * index should indicate not just that workload is varying but the cause of variation (Un)obstusiveness * an index should not interfere with or contaminate the primary task being assessed Reliability (Reproducibility) * index should produce the same estimate for a given task and operator Bandwidth * the index should respond to high-frequency changes in workload

Answer 29

1) primary task: direct 2) secondary task: indirect 3) physiological correlates 4) subjective ratings: does not interfere with task, but subjective

Answer 30

Measure performance metrics: * time * speed * strength Derived workload metrics * no absolute value * difference in performance my indicate difference in workload

Answer 31

popular types: * rhythmic tapping task * random number generation * probe reaction time task * time estimation * time production

Answer 32

* Heart rate (ECG), Muscle Activity (EMG), Brain Activity (EEG) * Respiration (GSR) * Oxygen uptake * Eye-Tracking In principle precise, but * difficult to set-up * needs extensive physiological conditioning to bring subjects to same conditioning level * different to compare between subjects due to high variation in physiological condition Conditioning: * Baseline Phase * Interaction Phase * Recover Phase

Answer 33

Mental demand * how mentally demanding was the task? Physical demand * how physically demanding was the task? Temporal demand * how hurried or rushed was pace of the task Performance * How successful were you in accomplishing what you were asked to do Effort * How hard did you have to work to accomplish your level of performance Frustration * How insecure, discouraged, irritated, stressed and annoyed were you?

Observation Flashcards

(57 cards)