Block 4 - Unit 2: Usability testing and field studies Flashcards Preview

M364 Revision > Block 4 - Unit 2: Usability testing and field studies > Flashcards

Flashcards in Block 4 - Unit 2: Usability testing and field studies Deck (54)
Loading flashcards...

Usability testing. (4)

Involves users.

Emphasises 'usable' property - product is being tested rather than user.

(Often) controlled environment; performance of users on pre-planned tasks is repeatedly measured.

Goal - test if product is usable by intended user population to achieve the tasks designed for.


Key components of usability testing. (2)

User test.

User satisfaction questionnaire.


User testing. (Include examples)

Measures human performance on specific tasks.

Eg. reading different typefaces, navigation of menu types, info searching.

Logging of mouse / keyboard and video used to record performance.


User satisfaction questionnaire.

How do users feel about using the product - rating along different scales, after interaction.


Measures used in usability testing.

Time and number:

- time to complete a task (after a specified time away);

- Number of errors per task or per time unit;

- Number of views of online help / manuals;

- Number of users making a particular error;

- Number of users completing a task successfully.


Number of users needed in usability testing.

5 - 12 considered acceptable.

Can use less if lower budget / schedule, or for quick feedback on eg. logo placement - 2 or 3 users.


Remote usability testing (including advantage).

Users perform tasks in their own setting, logged remotely.

Advantage - many users tested at once, and logged data automatically compiled into statistical packages for analysis. Eg. no of clicks per page.


Usability testing (UB)

Doesn't have to result in accurate measures of user performance, or be under controlled conditions, but does involve some form of performance assessment and a level of evaluator control, most commonly the use of pre-set tasks.

Several methods may be used - eg. indirect observation, user test to measure performance, questionnaire.

User tests often conducted along experimental lines, involving hypothesis testing and statistics.


Doing user testing. (Intro)

How to do user testing is peculiar to usability testing.

Controlling test conditions is central - careful planning; ensure all material is prepared, conditions are the same for each user, what is being measured is indicative of what is being tested and that assumptions are made explicit in the test design.


Elements of doing user testing. (4)

Design typical tasks.

Select typical users.

Prepare testing conditions.

Plan how to run the tests.


Design typical tasks (user testing).

Appropriate task to test users' performance is critical.

'Completion' tasks set, eg. find website, create spreadsheet.

Quantitative performance measures obtained.

User tests most suitable for hi-fi prototypes, simulations and working products.
Task type depends on system type and evaluation goals / questions.
Eg. whether paper-prototypes, simulation or limited part of system's functionality will influence breadth and complexity of tasks set.

Generally, tasks are 5 - 10 mins and designed to probe a problem.
Often straightforward, but occasionally more complex, eg. join an online community or solve a problem.


Select typical users (user testing)

What characteristics?

Some products aimed at eg. old / young, novice / expert.

Equal male / female split, unless aimed at one.

Previous experience with similar systems?

If user population is large - short questionnaire can help identify testers.


Prepare testing conditions (user testing)

Environment controlled to prevent unwanted influences and noise that distorts results.


Plan how to run the tests (user testing).

Schedule an script, equipment tested, pilot test.

Start with easy task to build confidence and familiarisation.

Avoid long tasks; keep session under 1 hour.

Don't create too much data to analyse.


Potential sources of bias (user testing)

if don't match profile, may get misleading results.

Evaluation tasks:
If users get unintentional clues from tasks, this may prevent exposing UPs.

Test setting:
Lab is artificial, and may need conditions adding to more closely simulate actual environment of use.

Evaluator / observer bias:
Behaviour may be altered when being watched. Observer may forget they shouldn't ask leading questions, or help if user gets stuck.

'Think-aloud' alters behaviour.
Avoid 'order effects' - trying one design may give 'practice' to trying another. (So, counterbalance).

Reporting / analysis:
Data analysis and interpretation will be undertaken based on evaluator's knowledge and experience - so biased to some extent.


Field studies (intro)

Typically used to find how a product / prototype is adopted an used in people's working / everyday lives.

Such settings are 'messy' - activities overlap and constantly get interrupted.

The way people interact with products is often different than in a laboratory setting; a better sense of the success of a product is gained in the real world.


Trade-off of field studies.

Can't test specific hypothesis about an interface or account, with the same degree of certainty, for how people react or use a product (than in a lab).

So, harder to determine what causes certain behaviour or what is problematic about the usability of a product.

Instead, qualitative accounts and descriptions of people's behaviour and activities are obtained to reveal how they used the product and reacted to the design.


Some characteristics of field studies. (3)

Can be minutes, months or years.

Primary data collection - observing and interviewing people; video, audio and field notes.

May ask for paper or electronic diaries to be filled in at points in the day - when interrupted, encounter a problem or are in a particular location.


Things to do for field studies. (4)

Important to inform people of the length of study / session.

Need to agree part of site to be recorded and how.

Need to set up cameras unobtrusively, and if eg. in someone's home - how to turn on / off.

What happens if product breaks?


Planning an observational evaluation (intro).

Expand DECIDE to present a procedural approach to 'doing' observational evaluation.

Primary focus - evaluation of low-fi prototypes (especially card-based and interface sketches).


Kinds of task to consider ('I')

Core tasks (frequent)

Those very important to users or the business.

Those that have some new design features / functionality.

Critical tasks.

Ones you feel have to be validated with users for greater clarity and understanding of the design team.

Those that scenarios were based on, which you used for developing the conceptual design of the UI.


How to choose tasks.

Choose tasks that help in validating usability or UX goals, or that focus on any particular design features you want to assess - metaphor suitability, task flow, icon design, etc.
But, depends on prototype as to what tasks you can evaluate.


Other materials for evaluation. (9)

1. Briefings for evaluators.

2. Introductory material for participant.

3. Background info for participant.

4. Pre-session questionnaire or an interview plan.

5. Permission to gather and keep data.

6. Task cards.

7. Data collection forms.

8. Post-session interview plan or questionnaire.

9. Data analysis, interpretation and recommendation forms.


First steps before evaluation session.

Pilot study.

Revising any procedures and materials as required.


Steps in an observational evaluation session.

Greet user, brief them about the study / equipment and obtain informed consent.

Conduct pre-session questionnaire / interview.

Show prototypes. If low-fi, explain their rough nature.

Give first task; check understanding.

Give each of the other tasks.

Observe while each task is attempted - what they do, or say if 'think-aloud'. Record observations on paper / data collection forms.

Explore as appropriate other concerns / aspect of design - suitability of metaphor, task flow, icon design, structure, layout, tools, etc.

Conduct post-session questionnaires / interviews (if appropriate).

Thank user, and de-brief with post-evaluation discussion.
If others present, have group discussion - gives people chance to share their impressions with designers.

If evaluation is done with colleagues, can be useful to compare notes.


What are paper-prototypes good for exploring? (6)

Concepts and terminology.

Navigation, work and task flow.


Documentation / help.

Requirements / functionality.

Interface layout.


Comments on script for evaluating paper-prototypes.

Add any questions thought of to script.

Where possible cluster questions around relevant task, so you're exploring at one time as many of the issues related to the task being attempted as you can.

For aspects (eg. UX goals) that may not cluster - ask separately.

Take care not to ask leading questions - we want users' own opinions.


Quantitative and qualitative data (analysis)

Quantitative - data in the form of numbers, or easily translated to numbers.

Qualitative - data is difficult to measure, count or express in numerical terms in a sensible fashion.

Fallacy that certain forms of data gathering will only result in quantitative data and others in qualitative - all forms previously discussed can result in either.

Quantitative analysis users numerical methods to ascertain magnitude, amount or size of something.

Qualitative analysis - nature of something, represented by themes, patterns an stories.


Use and abuse of numbers.

Often unwarranted to turn qualitative data into numbers to manipulate and interpret with respect to goals, although people tend to believe numbers offer strong / clear conclusion.

Better to use % for 10+ only, but still better to use raw numbers to make understanding clear.


Initial processing of interview data.

Raw data - audio or notes.

Notes - write up and expand while memory fresh.
Audio can help, or may be transcribed (large effort - maybe just relevant parts).

Closed questions - quantitative analysis, eg. % in age range.

Open questions - qualitative analysis.