Quantitative Research Methods

Question 1

What is Quantitative Research?

Answer 1

Quantitative research in UX focuses on collecting numerical data to measure what is happening, how often, and to what extent. It is highly structured and often used to validate patterns across large sample sizes for statistical reliability.

The study of what can be measured and observed.

The results will be consistent and generally agreed on by all parties involved.

Question 2

What are some examples of quantifiable and qualitative human traits?

Answer 2

Height and weight are quantifiable measurements, as they can be counted and measured against a standard scale.

Personality traits like “nice” or “funny” are subjective.

Question 3

What are quantitative measurements?

Answer 3

• Bounce rates
• Time on task
• Conversion rates
• Order size (number of items or their value)
• Number of visitors to a site (physical or digital)
• Average size of group

Question 4

What are the most common aspects of quantitative research and what do these common aspects help with?

Answer 4

Mean (average value), median (middle value), mode (most common value), and range (difference between the highest and lowest values)

They are helpful for a high-level understanding of averages and trends.

Question 5

What is a major limitation of quantitative research?

Answer 5

It doesn’t tell us how to fix things, why things happen, or share information that isn’t specifically asked for.

Quantitative UX research alone does not explain the why behind issues—it only shows the what and how often.

It lacks human context and does not capture intent, emotion, or environment.

Question 6

Despite its limitations, how can quantitative research be useful?

Answer 6

It can act as a benchmark for future studies and for qualitative research.

Question 7

What does quantitative research (e.g., analytics, usage metrics) tell you?

Answer 7

It tells you what is happening.

Question 8

What kinds of insights does quantitative research show?

Answer 8

It shows behavior patterns, frequency of events, drop-offs, etc.

Question 9

What can happen if you rely only on quantitative metrics?

Answer 9

Metrics can be misleading without understanding how and where a product is being used.

Fixes based solely on numbers may be superficial or entirely misdirected.

Question 10

Why should quantitative data always be complemented with qualitative methods?

Answer 10

To understand the root cause and context of user problems.

Question 11

What are the three main focuses of research that inform product design work?

Answer 11

Insight-driven, evaluative, and generative research.

More commonly called planning, discovery/exploration, and testing/validation.

Question 12

How do these research focuses impact product design beyond initial ideation?

Answer 12

They influence maintenance and support goals for ongoing projects.

Question 13

Is design and iteration finished once a product ships?

Answer 13

No, design and iteration continue after launch, especially with the ease of pushing updates in digital products.

Question 14

What mindset should product teams maintain after shipping a product?

Answer 14

Always be looking to measure and improve the products delivered.

Question 15

What does insight-driven research seek to understand?

Answer 15

It seeks to understand what the problem space is, why the problem exists, and where opportunities lie.

It’s not about measuring how a solution performs, it’s about discovering what problem to solve and what success could look like.

Question 16

When is insight-driven research typically conducted?

Answer 16

In the early stages of projects.

When it’s used:
- Before solutions are designed
- When you’re skill learning about users, behaviors, pain points, or market gaps
- To identify opportunities and unmet needs

Question 17

What is a simple example of insight-driven research?

Answer 17

Looking at the rate at which users succeed when attempting to accomplish their own goals.

Question 18

How does insight-driven research appear in quantitative research?

Answer 18

As benchmarks and initial KPIs (e.g., success rates, task completion, drop-off points)

Question 19

What are some insight-driven research methods?

Answer 19

• Qualitative interviews
• Diary studies
• Contextual inquiries
• Behavioral analytics (early signal exploration)

Question 20

What is the purpose of evaluative research?

Answer 20

To assess how well a solution works against goals, benchmarks, and KPIs that were identified.

To measure how a design or solution performs against established KPIs and benchmarks.

Question 21

When is evaluative research conducted?

Answer 21

• During and after design implementation
• To validate if solutions are usable, useful, and effective
• To refine and iterate on designs21523

Question 22

What is the quantitative benefit of Evaluative Research?

Answer 22

Can show improvements at scale (e.g., % reduction in errors, time on task)

Question 23

What does evaluative research help determine about a user flow?

Answer 23

How effective the flow is, both before and after proposed changes.

Question 24

Can evaluative research use both quantitative and qualitative methods?

Answer 24

Yes, it may include both types of measurements.

Question 25

What advantage does quantitative evaluative research provide?

Answer 25

It can pinpoint opportunities for improvement on a larger scale than qualitative research.

Question 26

What is the relationship between Insight-driven research and Evaluative research?

Answer 26

Insight-driven -> uncovers what to solve Evaluative -> measures how well you solved it One informs and supports the other. They are complementary, not competing.

Question 27

What is Generative Research?

Answer 27

(also called Data-Driven design) helps create and inspire new ideas, concepts, and design directions through research. It's used to understand users deeply so that teams can generate solutions that are grounded in real needs, behaviors, and motivations. help UX teams explore new design possibilities by uncovering deep user needs, goals, and pain points. This research fuels data-driven design by blending objective insights with measurable patterns or trends. Though primarily qualitative, generative research can be informed by quantitative data to identify areas of opportunity. It focuses on creating the right solution—not just testing an existing one.

Question 28

At what stage of the design process is generative research conducted?

Answer 28

In the early phase, before defining solutions.

Question 29

What Are System Analytics?

Answer 29

System analytics (or site analytics for web experiences) are one of the most common forms of quantitative UX research. These tools provide passive, ongoing data collection about user behavior, system usage, and interaction flows.

Question 30

Why are System Analytics valuable?

Answer 30

- Low cost of entry: Many platforms like Google Analytics, Mixpanel, or Hotjar offer free or low-cost plans. - Passive data: Once properly implemented (i.e., correct event tagging and tracking setup), they run in the background, collecting data at scale. - Breadth of data: They provide large-scale insights that are difficult to gather manually.

Question 31

How does System Analytics support Insight-driven research?

Answer 31

- Highlighting where users struggle - Revealing usage patterns and drop-off points - Surfacing unexpected behaviors at scale - Informing hypotheses for further qualitative investigation

Question 32

What are user flows?

Answer 32

Paths users take through your product.

Question 33

How are user flows used in the context of system analytics?

Answer 33

To identify navigation patterns, drop-offs, or shortcuts.

Question 34

What are Entry Points?

Answer 34

How users access your experience (homepage, search, direct link, etc.)

Question 35

How are Entry Points used in the context of system analytics?

Answer 35

Understand product discovery and funnel efficiency

Question 36

What are Search Logs?

Answer 36

Terms users input in your search field

Question 37

How are Search Logs used in the context of system analytics?

Answer 37

Reveal natural mental models, unmet needs, and poor navigation/taxonomy

Question 38

What are Demographics?

Answer 38

Age, gender, location, device, etc. (depending on consent and data availability)

Question 39

How are Demographics used in the context of system analytics?

Answer 39

Ensure product aligns with target user segments

Question 40

What are Surveys?

Answer 40

Surveys are proactive research tools used to gather direct input from users. Unlike analytics, which passively collect data based on user behavior, surveys allow you to ask users directly about their intentions, needs, experiences, and satisfaction.

Question 41

Why Surveys Are Versatile?

Answer 41

Surveys are uniquely positioned because they can serve both: 1. Insight-driven: Learn about user expectations, goals, intent, unmet needs. 2. Evaluative: Assess satisfaction, usability, NPS, post-task impressions, CSAT, etc.

Question 42

What are common Survey Goals?

Answer 42

1. Understand user intent "What brought you here today?" 2. Evaluate Usability: "How easy was it to complete this task?" 3. Measure Satisfaction: "How satisfied are you with your experience?" 4. Assess expectations vs. Reality: "Did this experience meet your expectations?" 5. Capture Net Promoter Score (NPS): "How likely are you to recommend us to a friend or colleague?"

Question 43

What are the key strengths of Surveys?

Answer 43

1. Direct user voice: You're not guessing based on behavior, you're hearing it firsthand. 2. Scalable: Can be deployed broadly for statistical significance. 3. Targeted: Can be triggered by specific actions or events (e.g., cart abandonment, feature usage).

Question 44

Let's say analytics show 60% of users abandon the checkout page. Analytics tell you: There's a problem. What can a post-abandonment survey reveal?

Answer 44

It can uncover the reason why users abandoned the checkout—such as unexpected costs, confusing forms, or lack of trust—providing context that analytics alone cannot show. Together, these methods give you both the “what” and the “why.”

Question 45

What Is Tree Testing? aka, Tree Jacking or Reverse Card Sorting

Answer 45

Tree testing is primarily an evaluative UX method used to measure how users navigate and understand a proposed information architecture or taxonomy. Designers input a hierarchy into a testing tool, and users are asked to complete tasks by selecting where they'd look for information. The data collected—such as success rates and click paths—helps identify where the structure aligns with or deviates from user expectations. While not inherently generative, tree testing can inform new navigation structures through repeated iteration.

Question 46

How does Tree Testing work?

Answer 46

1. A designer or researcher inputs the proposed hierarchical structure (i.e., site map or menu) into a tree testing tool. 2. Users are given realistic tasks (e.g., “Where would you go to change your billing address?”). 3. Users complete the tasks by navigating through the text-based tree structure—no visual design or UI distractions. 4. The system tracks: - Success rates - Directness of path - Time to complete - Drop-offs or wrong turns

Question 47

Why is tree testing considered evaluative in nature?

Answer 47

Because it measures how well users understand and navigate an existing or proposed structure.

Question 48

How can tree testing also inform generative decisions?

Answer 48

When combined with other insights, it can guide how to redesign a taxonomy.

Question 49

What Is Eye Tracking?

Answer 49

Eye tracking is a method of measuring where and how users look at a screen while interacting with a digital interface. It uses specialized cameras and software to track a participant’s eye movements in real time, capturing: 1. Fixations (where the eyes stop) 2. Saccades (quick jumps between fixations) 3. Scan paths (the visual journey across the interface) 4. Heatmaps (aggregated visual attention across users)

Question 50

For what types of screen-based digital products is eye tracking especially valuable?

Answer 50

E-commerce websites, landing pages, product detail pages, navigation-heavy platforms, and interactive dashboards or tools.

Question 51

What does eye tracking reveal in UX research?

Answer 51

Visual hierarchy effectiveness, CTA visibility, label clarity, distracting or ignored elements, and the difference between user attention and the intended design path.

Question 52

What Is A/B Testing?

Answer 52

A/B testing is a controlled experiment in which two or more versions of a webpage, screen, or element are shown to users at random. The goal is to compare performance and identify which version performs better based on predefined KPIs. It falls under site/system analytics as a quantitative evaluative method.

Question 53

How does A/B testing work?

Answer 53

1. Create a hypothesis 2. Design two versions - Version A: Original (Control) - Version B: Modified (Variant) 3. Split users randomly 4. Measure KPIs 5. Analyze Statistical Significance (Use tools like Google Optimize, Optimizely, or VWO to determine if differences are meaningful.)

Question 54

What are the benefits of A/B testing?

Answer 54

- Data-driven decision making - Low-risk experimentation - Micro-optimization of UI/UX - Easy to implement with tools

Question 55

What are limitations of A/B testing?

Answer 55

- Retrospective: A/B testing only tells you what worked, not why it worked - No user intent captured: It lacks insight into user motivations, emotions, or unmet needs - Required high traffic: For meaning results, especially for small changes - Single-variable focus: Complex UX problems often involve multiple layers that A/B can't isolate - Can lead to local optima: You may optimize a small part of the experience without improving the whole

Question 56

What is A/B testing ideal for?

Answer 56

Testing micro-interactions like CTA wording or image choice.

Question 57

What must guide effective A/B testing?

Answer 57

A clear hypothesis and alignment with established KPIs.

Question 58

What limitation does A/B testing share with other system analytics?

Answer 58

It provides a backward-looking view of user behavior and does not capture user intent or subjective experience.

Question 59

What Is Card Sorting?

Answer 59

Card sorting is a generative UX research method used to understand how users categorize information. Participants are asked to group topics, labels, or features into logical categories based on their own understanding.

Question 60

What does card sorting help designers and researchers do?

Answer 60

Create or refine information architecture, define navigation menus, improve labeling (taxonomy), and uncover user mental models.

Question 61

What are the three types of card sorting?

Answer 61

1. Open card sorting: Participants create their own categories and group items freely 2. Closed Card Sorting: Participants sort cards into predefined categories provided by researcher 3. Hybrid: Combines both open and closed- Users can use existing categories or create new ones.

Question 62

What does Card Sorting reveal?

Answer 62

1. Natural language use 2. User expectations 3. Preferred groupings 4, Inconsistencies in content structure or labels.

Question 63

What is a Moderated product testing?

Answer 63

Conducted in-person or live online with a facilitator. Enables deeper probing into decisions.

Question 64

What is a Unmoderated product testing?

Answer 64

Validating a product with a researcher setting up questions for a customer to respond to at a later time. Mostly done remotely using digital tools

Question 65

What is Intercept Testing?

Answer 65

A method of randomly requesting customer feedback as they engage with a product

Question 66

When are quantitative research methods most effective?

Answer 66

When a large number of participants or customers are assessed for statistically significant outcomes.

Question 67

Can quantitative methods be useful in smaller studies?

Answer 67

Yes, methods like eye tracking and card sorting can show value with only a handful of participants.

Question 68

When are quantitative methods best used?

Answer 68

When the research question has a tangible, measurable outcome. - You need statistically significant insights. - You’re evaluating behavior at scale. - You're trying to measure performance, trends, or changes over time. - Your research question has clear, measurable outcomes.

Question 69

Example Research Questions for Quantitative Methods:

Answer 69

“What percentage of users abandon the checkout process?” “How many users click on the ‘Learn More’ button?” “What is the average time spent completing a task?” “How does Version A of a landing page perform compared to Version B?” These types of questions are ideal because they are objective, countable, and can be tied directly to UX KPIs (e.g., conversion rate, completion time, error rate).

Question 70

What are some key shortcomings of quantitative methods?

Answer 70

They may not effectively capture user motivation or task comprehension. 1. Lacks context and motivation: It tells you what happened, but not why. It doesn't capture user emotions, intent, or decision-making processes 2. Required sufficient data: Tools like analytics and A/B testing rely on large sample sizes for statistical significance. With limited users, results can be misleading or inconclusive. 3. Not helpful in early-stage innovation: If you're developing a product for a new market, there may be no behavioral data to analyze yet. In these cases, generative or qualitative methods are more effective 4. Doesn't reveal comprehension or usability pain: If users drop off during a process, analytics might show where they dropped, but not whether it was due to confusion, frustration, or unmet expectations. 5. Hard to explore unknowns: Quantitative methods measure what you already know to look for. They don't uncover new problems, unmet needs, or hidden opportunities the way open-ended methods do.

Question 71

When are qualitative methods more effective than quantitative methods?

Answer 71

When trying to understand user motivations, intentions, and how users comprehend a task.

Question 72

When should quantitative methods be complemented by qualitative research?

Answer 72

1. When you need to explore user motivations or thought processes. 2. When you're building something new with little existing user data. 3. When you have limited access to users or small sample sizes. 4. When you're exploring early concepts, designs, or task comprehension.