Quant Methods and Measurement

Question 1

Briefly describe quantitative methods

Answer 1

• Often apply statistics to data, analyse text and documents or as an approach to observation
• Methods include standardised questionnaires and surveys
• Relies on manipulation of statistical data
• Concentrates on generalities, normality or departures from it

Question 2

Distinguish between the different levels of measurement/measurement scales (provide examples)

Answer 2

Nominal - classifying and labelling two or more categories without making assumptions about their relative value. Compare and test distribution.
Example: diagnosis or occupation

Ordinal - labels categories and ranks them in order.
Example: level of patient dependency/satisfaction

Interval - Not only ranks categories of interest but does so in a way that there is known difference between them. Arbitrary starting point.
Example: temperature

Ratio - Ranks with absolute starting point of zero so distance between points can be compared proportionally or by ratio. Strongest measurement.
Example: Cost

Question 3

What role does theory/methodology play in measurement?

Answer 3

Influences how indicators are chosen due to the abstract nature of concepts measured in the social science (in comparison to science which have clear units of measurement and instruments)
Appropriate methods in social science are more complex and are often contested.

Question 4

Explain reliability in quantitative research

Answer 4

Measures is free from undesirable noise or interference and is able to measure “true differences” without error

Established via:

• Test-retest reliability - instrument is stable, respondents complete the same form at two different times with no change expected
• Inter-rater/observer validity - instrument is tested to ensure that different raters observe the same way
• Internal consistency - items in instrument are contested with each other, tested statistically

Question 5

Explain validity in quantitative research

Answer 5

How well chosen indicators actually measure the underlying concept.
- A single indicator is rarely adequate (eg. WHO concept on health = physical, mental and social

Cannot be proven but methods to demonstrate or strengthen credibility include:
- Face validity: “looks like” measure of variable of interest (subjective)
- Content validity: representativeness of a range of indicators that cover all elements of the concepts eg. wide range of symptoms for patients undertaking chemo for cancer
- Criterion validity: indicator chosen correlates with other “known” indicators eg. blood alcohol level
- Construct validity: generates hypotheses to test if indicator performs the way the theory says it should
Eg. known groups - patients with advanced cancer to have lower quality life scores than those in early stages;
convergence- expectation of scores to correlate positively with known indicators;
non-convergence - not expected women to score differently to men (unless part of theory)
Key issue here is ability to see if the problem is with the instrument or the theory itself

Question 6

Describe questionnaires as quantitative instrument (including the advantages and disadvantages or their use)

Answer 6

Questionnaires are a structured schedule to elicit quantitative data

Types:
Survey - questions are analysed individually
Psychometric - items are combined for an overall score

Methods: person, phone, postal, internet

Advantages:

• Cost-effective (for both distribution and analysis)
• Easy to distribute;
• Easy to analyse (due to consistency, comparability and generalisation)
• Standardised to potentially reduce bias
• Less time-consuming and intrusive for participants

Disadvantages:

• Low response rates
• Missing or incorrect answers
• Inflexible
• Cannot probe (limited depth
• Error and bias

Question 7

Identify some common issues when designing a questionnaire

Answer 7

Excess mental demands

• misunderstanding a question
• inability to recall
• guessing
• mapping onto alternatives

Biased responses

• sacrificing (providing satisfactory rather than optimal answer
• social desirability (faking good)
• deviation (faking bad)
• acquiescence (agree rather than disagree)
• end avoidance
• positive skew

Question 8

Describe some strategies for minimising bias

Answer 8

• Provide a clear rubric and instructions
• Clear and specified time frame
• Mix of positive and negative questions (to avoid faking good/ad)
• Control for social desirability
• Use non-absolute end points (almost always) or throwaway responses (does not apply) to minimise aversion
• Use behaviourally anchored scales to reduce halo effects
• Place items likely to influence each other separately to reduce framing

Question 9

Describe the process of questionnaire design

Answer 9

Step 1: Develop a measurement strategy

• What to measure? (risks, variables, outcomes, knowledge behaviour, attitudes)
• Type (survey or psychometric)
• Administration (self, interview, observer, records)
• When/how often (single or repeated)
• Measure culturally appropriate?

Step 2: Develop a conceptual framework of the construct to be measured

• Literature review
• Review of existing instruments
• Interviews and focus groups
• Expert opinions

Step 3: Develop the content and format of the questionnaire

• Questions developed from conceptual framework
• Decide response format and scaling
• Plan sequence, appearance and layout
• Test

Question 10

Name some key considerations for good questionnaire design

Answer 10

Questions:

• Clear language and questions (simple, neutral, direct)
• Produce variability in response
• Appropriate range and design of answers - accomodate possible answers, don’t imply response, mutually exclusive, non ambiguous

Questionnaire:

• Adequate space for responses
• State clearly purpose of research and who is conducting it
• Policy on confidentiality and anonymity
• Clear instructions
• Begin with interesting and on threatening items
• Put most interesting items first
• Group questions into coherent categories
• Professional production methods
• Express gratitude
• Easy completion and return
• Piloted

Question 11

Describe cross-sectional surveys as quantitative instrument (including the advantages and disadvantages or their use)

Answer 11

Used for data collection or study in which standardised variables are measure across a sample at one point in time

Advantage - produces standard set of quant data that can be analysed statistically for patterns, regularities, relationships
Disadvantage - acceptability of method, can be threatening, dishonesty
Key issue of concern is generalisability -> this stems from representativeness of the sample chosen

Question 12

Explain the main two approaches to sampling

Answer 12

Probability sampling - each member of population has a known probability of selection. Sampling error can be calculated.

Non-probability sampling - any sample not randomly drawn from the population. Sample error is unknown and is likely to contain bias

Question 13

Explain the different types of probability sampling

Answer 13

• Simple random sample: whole population of interest used as sample frame, equal chance of selection eg. registers, postcodes etc.
• Systematic sampling: every nth record selected from list of population members to required sample size. Simple but can be bias in list.
• Stratified sampling: random sampling used to pick representative number of subjects from each strata (subset of population that share a common characteristic). Reduces random sampling error.
• Cluster sampling: sampling of complete groups of hierarchical units eg. household. Convenient but increases sampling error.

Question 14

Explain the different types of non-probability sampling

Answer 14

• Purposive sampling: deliberate choice of respondent, subject or setting for characteristic of interest
• Convenience sampling: samples selected as easily available
• Snowball samples: networking out from purposive and convenience samples to reach other less accessible subjects
• Quota samples: allocated according to proportional distribution of different demographic characteristics (that can be known). Sample within each quota not randomly selected so different to stratified sampling.

Question 15

What are some of the ways that survey studies can be more representative?

Answer 15

• Sample size -> larger size generally more reliable; need the prevalence and variation of target event/behaviour
• Sample responsiveness -> higher response rate = greater confidence in results. Low response rate can cause bias (generally 70%). Rewards and incentives can be useful but also can create bias.
• Sample representativeness-> comparing structure of sample and population via demographic information (other variables often hard to measure). Difference in participation vs. non-participation important to the study.

Question 16

What are the different methods of administration for surveys? What are some of the advantages and disadvantages of each?

Answer 16

In-Person
Advantage - Longer interaction possible, more flexible (can probe), can use more complex questionnaire
Disadvantages - More costly, takes longer, interviewer bias, respondents less honest about sensitive issues

Phone
Advantage - faster, less costly, respondents may be more honest
Disadvantages - limited time window, may be inconvenient, not all have landlines, less honesty if others present

Postal
Advantage - less costly, convenient for respondent, eliminate interviewer bias, may be more honest
Disadvantages - longer than telephone (faster than fact to face), response rate may be low, less control over completion
(If sent by email similar but faster, resistance to “spam” and who has access to internet

Internet
Advantage - Less costly, fast (to perform and analyse), can use features eg. stop skipping, can be attractive, eliminates interviewer bias, honesty, tools available
Disadvantages - representativeness of respondents, internet more casual and optional, low response, difficult to get a probability sample

Question 17

Explain the concepts of sampling and non-sampling error

Answer 17

Sampling Error: Due to an unrepresentative sample, inferences cannot be generalised to the whole target population. Stems from a small sample size or invalid methods. Can often be identified and controlled for.
EXAMPLE: Target population is a whole community but sample does not include minorities or hard to reach groups, favours a particular gender, age, socio-economic status etc.

Non-Sampling Error: Errors in collected data that stems from problems with validity and reliability the data collection method. Doesn’t measure what it purports to measure or too much noise in instrument. More difficult to control for, can sometimes be reduced by qualitative work and pre-testing.
EXAMPLE: Social desirability impacting how respondents answer questions about contraceptive use

Question 18

Identify ways in which survey sampling can be more representative

Answer 18

• Sample size -> larger sample size generally more reliable; need prevalence and variation of target behaviour
• Sample responsiveness -> higher response rate = greater confidence in results. Low response rates can cause bias (generally 50-70%). Rewards and incentives can be used but also create bias.
• Sample representativeness -> comparing structure of sample and population via demographic information. differences in participation vs. non-participation are important for study findings.

Question 19

What are some of the limitations of surveys?

Answer 19

• acceptability of form filling not universal
• can be threatening
• responder bias - re-call and social desirability