Week 4 User Study Flashcards

Question

Physiological Data

Answer 1

Eye-tracking Heart rate, breathing rate ECG (cardio), EEG (brain) Skin conductance Motion Requires specialized equipment

Answer 2

Laboratory Field Web Extra: Crowd-Sourced Study

Answer 3

Characteristics * Controlled environment * Direct observation Used for * Measuring performance & improvements * Discovering usability problems * Gaining confidence of design before riskier testing Potential Drawbacks * Realism can be difficult to achieve * Novelty effects (hey, this is cool!)

Answer 4

Characteristics * Deployed in regular setting * Maybe more ecologically valid * More expensive and time-consuming Used for * Evaluate real-world usage * Gather long-term data Potential Drawbacks * Riskier for security, privacy, reliability perspectives * Influenced by outside factors, unsupervised * Data much more “messy

Answer 5

Characteristics * More natural setting * Uncontrolled (realistic) environment * Can prompt to complete tasks, screenshare, videoconference Used for * More convenient than lab studies * Middle ground between lab and field studies Potential Drawbacks * More difficult to observe * Technical challenges

Answer 6

Characteristics * Fast and cheap data collection through online crowdsourcing Used for * Quickly gathering large amounts of data * Studies that can be broken into small tasks Potential Drawbacks * Data quality/reliability * Need fully automated study protocol

Answer 7

Explores the same topic in multiple ways * to gather more holistic understanding, increase validity, credibility Uses multiple * Data sources (people, places, times) * Data collection methods * Evaluators Potential Drawbacks * Time-consuming data analysis

Answer 8

Goals & questions focus on how well users perform tasks with the product Comparison of products or prototypes is common Focus is on time to complete task, number & type of errors Data collected by video & interaction logging Testing is central Satisfaction questionnaires & interviews provide data about users’ opinions

Answer 9

Predict the relationship between two or more variables Independent variable is manipulated by the researcher Dependent variables are what you are measuring Validated statistically & replicable

Answer 10

Between-Subjects & Within-Subjects

Answer 11

Each participant is tested on one condition only Pros * No ordering or training effects Cons * Need more participants, * Chance of variations between subjects

Answer 12

Each participant is tested on all conditions Pros * Need fewer participants * Can see variation in performance across conditions per participant * Eliminates variation of participants between conditions Cons * Ordering effects * Training effects

Answer 13

Type of study Evaluation measures, conditions, instrumentation Tasks, scenarios Scripts and instructions Questionnaires, Interview scripts Hardware and software

Answer 14

Be specific and realistic Use scenarios if possible Ask user to complete tasks

Answer 15

Instructions matter, can easily bias user behavior Users help evaluate/test thesystem You are grateful for any feedback – good or bad!

Answer 16

Screen recording (e.g., Camtasia) Noldus Observer, Morae Observer Eye-tracking, physiological measures Video

Answer 17

Log user actions, timing info Test for accuracy and format of logs! Include testing-specific prompts and messages

Answer 18

Representative users How many? 5-10, 20+ for significant experiment Remuneration * What is reasonable? * Do not want to influence behavior

Answer 19

An evaluation can be a distressing experience Participants should always be treated with respect

Answer 20

Don’t waste user’s time Make users comfortable Maintain privacy Inform user Users must volunteer

Answer 21

Any research involving humans must be cleared Document the study protocol Document the purpose of study Submitted to the Research Ethics Board

Answer 22

Work backwards from your goals to figure out what you need to do: What are your overall goals at this point? What are the questions you want to answer? What kinds of data do you need to collect? What type of study fits these criteria?

Answer 23

What are your overall goals at this point in the process? May be an iterative process May have goals at different levels

Answer 24

What are your overall goals at this point in the process? Current study goals What kinds of questions address these goals?

Answer 25

Qualitative, Quantitative, Performance, all pertaining to specific needs for the type of study. As in what questions do you need to ask to get the data

Answer 26

What are the types that fit? Field, Lab, Web, Crowd Source

Answer 27

On what do you want to focus? Make a specific plan for the user study * Type of study (considering time, resources, expertise) * Protocol * Instruments * Data analysis Realism? * Will results apply in real world? Generalizability? * Will results apply to other situations?