Research Methods & Study Design

Question 1

Correlation is not…

Answer 1

Causation

Question 2

What are the possible causal relationships that could explain a correlation that is found b/w two variables, A & B?

Answer 2

◦ A -> B
◦ B -> A
◦ C -> B + A (separatly)

Question 3

What steps are needed for a good experimental design?

Answer 3

1) Select the population
2) Operationalize the independent and dependent variables
3) Carefully select the control and experimental groups
4) Randomly sample from the population
5) Randomly assign individuals to groups
6) Measure the results
7) Test the hypothesis
◦ A good experimental design often involves making compromises to meet conditions or to make conducting a study realistic

Question 4

How do you select the population?

Answer 4

◦ Agree on a population of interest
◦ Find a way to incentivize/find willing participants
◦ Will the selected population lead to follow up studies in the future (this might be a needed result)

Question 5

What does it mean to operationalize the independent and dependent variables?

Answer 5

◦ Have to have very specific definitions of each variable - be aware that the operationalization of each variable my introduce limitations to the study
◦ The independent variable must be directly manipulated by the researchers
◦ Researchers want to have maximum control over the experimental environment so that they can be sure that differences b/w the groups actually led to the effect, assuming one is measured

Question 6

What is an independent variable?

Answer 6

The independent variable is the variable manipulated by the research team

Question 7

What is a dependent variable?

Answer 7

The variable that is measured

Question 8

What is the importance between the independent and dependent variables?

Answer 8

◦ To prove a causal relationship
◦ Researchers want to have maximum controlover the experimental environment so that they can be sure that difference b/w groups actually led to an effect (if one was observed)

Question 9

Define reproducability

Answer 9

Good experimental design requires experiments that can be reproduced by other researchers (which is an important quality of good experimental design)
◦ Important as other researchers may want to verify the results or adapt some aspect of the experiment

Question 10

What is an operational definition?

Answer 10

A specification of precisely whjat they mean by each variable

Question 11

What is a quanitiative variable?

Answer 11

Numberical (can be quanitified)
◦ This data will be necessary to conduct statistical analyses that will test the hypothesis

Question 12

What is a qualitative variable?

Answer 12

Descriptive (can be described) or categorical

Question 13

How can one carefully select control and experimental groups?

Answer 13

◦ Without both a control group and an experimental group, a study is not experimental and causal relationships cannot be drawn
◦ The control group has to be homogenous (the same throughout), and as similar as possible to the experimental group except for the variable of interest (the treatment)
◦ Identify what kinds of (extraneous) variables could affect the experiment -> want to select experimental and control groups that are as similar as possible to relation to those potential extraneous variable identified - they must be as similar as possible

Question 14

What is the experimental group?

Answer 14

The group of participants that receives treatment

Question 15

What is the control group?

Answer 15

The group of participants that don’t receive treatment and act as a point of reference and comparison for the experimental group

Question 16

What is the objective of having a similar control group?

Answer 16

To rule out extraneous (or confounding) variables

Question 17

What are extraneous/confounding variables?

Answer 17

◦ Variables other than the treatment that could potentially explain an experimental result
◦ Extraneous variables not accounted for in the study
◦ Another variable offers an alternative explanation for results
◦ Lack of a useful control

Question 18

What is the placebo effect?

Answer 18

The fact that by just believing that the treatment is being administered can lead to a measurable result

Question 19

What is a double blind study?

Answer 19

When neither the personadministering treatment, not the participants truly know if they are assigned to the treatment or control groups

Question 20

How does one randomly sample from the population?

Answer 20

◦ Sampling should be random - therefore, it should be equally likely for any member of the population to be a participant in the study

Question 21

What is sampling bias?

Answer 21

◦ It is not equally likely for all members of a population to be sampled
◦ Means selection criteria is not random
◦ Population used for sample does not meet conditions for statistial test (ex. population is not normally distributed)
◦ Can threaten internal validity

Question 22

What is selection bias?

Answer 22

A more general category of systemic flaws in a design that can compromise results
◦ Attrition is a type of selection bias

Question 23

What is meta-analysis?

Answer 23

A big picture analysis of many studies to look for trends in the data

Question 24

What is attrition effects?

Answer 24

◦ Participants dropping out of the study/participation fatigue
◦ If the reason that participants are dropping out is non-random, this might introduct an extraneous variable
◦ Can threaten internal validity

Question 25

How to randomly assign individuals to groups?

Answer 25

◦ It should be equally likely that individuals are assigned to either group ◦ May use the randomized block technique

Question 26

What is the randomized block technique?

Answer 26

Researchers evalute where participants fall along the variables they wish to equalise across experimental and control groups. Then they randomly assign individuals from these groups so that the treatment and control groups are similar along the variables of interest

Question 27

How are results measured?

Answer 27

◦ Those conducting the experiment must check to make sure that their measurements are valid (the most important aspects od measurment for an experiment are that the dependent variable is quantitative (and therefore measurable) and that the instruments used are reliable

Question 28

What is construct validity?

Answer 28

That something (an instrument, likely) measures what its supposed to

Question 29

What is replicability?

Answer 29

The repeated measurments lead to similar results

Question 30

What is psychometrics?

Answer 30

The study of how to measure psychological variables through testing

Question 31

What is response bias?

Answer 31

The tendency for respondents to not have perfect insight into their state and provide inaccurate responses (which can be a concern with surveys)

Question 32

What is a 'between-subjects' design?

Answer 32

The comparisons are made between subjects, from one group to another

Question 33

What is a 'within-subjects' design?

Answer 33

Where subjects within one group are compared at different points in time

Question 34

What is 'mixed-methods' research?

Answer 34

◦ Any combination of different research techniques, such as within-subjects and between-subjects, or qualitative and quantitative

Question 35

Define reliability for the instruments used in a study

Answer 35

It means that they produce stanle and consistent results, measure what they're supposed to, and that repeated measurents lead to similar results

Question 36

How does one test a hypothesis?

Answer 36

◦ Start with a null hypothesis (assume if any effects are measured, its simply due to chance) ◦ Only then, they see if the evidence from the experiment proves the null hypothesis to be true or false - this puts the 'burden of proof' on the experimental hypothesis ◦ Pick a value to demonstrate a 'significant difference' b/w the variables (usually 5% or 0.05)

Question 37

Which is prefered, a type 1 error or a type 2 error?

Answer 37

It is better to incorrectly conclude that there is no effect (a type 2 error), than to fasely conclude there is an effect, when one actually does not exist (a type 1 error)

Question 38

What is a type 2 error?

Answer 38

A false negative (to incorrect conclude there is no effect) ◦ Accepting the null hypothesis when it is false

Question 39

What is a type 1 error?

Answer 39

A false positive (to incorrectly conclude there is an effect) ◦ Rejecting the null hypothesis when it is true

Question 40

What is a null hypothesis?

Answer 40

The hypothesis with the assumption that there is no causal relationship b/w the variables and that any effect that is measured (if there is one), is due to chance

Question 41

What is an experimental hypothesis?

Answer 41

The hypothesis that argues that the variations in the independent variable cause changes in the dependent variable

Question 42

How can one reject the null hypothesis?

Answer 42

◦ One cannot simply observe a difference b/w two groups as the observed difference may simply be due to change ◦ Have to determine if a significant difference has been achieved b/w groups - this is usually a point at which it is reasonable to conclude 'beyond a reasonable doubt' that there was a difference (usually 5%, or 0.05)

Question 43

What is a 'significant difference'?

Answer 43

A measured difference b/w two groups that is large enough that is probably not due to chance

Question 44

What is a p-value?

Answer 44

A number 0 to 1 that represents the probability that a difference observed in an experiment is due to chance ◦ Conventionally, if p < 0.05, scientists reject the null hypothesis

Question 45

What is a sample size?

Answer 45

A number of participants participating in the study ◦ Those conducting the experiment generally want a large/larger sample size to ensure that the experiment picks up an effect (easier to do and is more accurate with a larger sample size)

Question 46

What happens if p < 0.05?

Answer 46

The null hypothesis is rejected

Question 47

What approx. number of participants is generally mathematically needed to conduct statistical tests?

Answer 47

30+

Question 48

Define power

Answer 48

The ability to pick up an effect during an experiment, if one is actually present

Question 49

What does a larger sample size do?

Answer 49

It increases the power of the experiment

Question 50

What is suggested by a lower p-value?

Answer 50

If met/achieved, it means there is a very strong relationship b/w the variables, and a very small chance of the null hypothesis being true (ie. if p < 0.001, if the null is true, it means there is less than a 1 in 1000 chance of observing the results)

Question 51

Good Experimental Design - Step 1. ◦ Objectives ◦ Common flaws in design

Answer 51

Step 1: Select the population Objectives: ◦ Determine the population of interest ◦ Consider what group will be pragmatic to sample Common flaws in design: ◦ Population is too restrictive ◦ Sampling all individuals of interest is not practical

Question 52

Good Experimental Design - Step 2 ◦ Objectives ◦ Common flaws in design

Answer 52

Step 2: Operationalize variables Objectives: ◦ Determine the independent and dependent variables ◦ Specify exactly what is meant by each ◦ Make sure the dependent variable can be measured quantitatively within the parameters of the study Common flaws in design: ◦ Insufficient rigor in description ◦ Manipulation of the independent variable presents practical problems

Question 53

Good Experimental Design - Step 3 ◦ Objectives ◦ Common flaws in design

Answer 53

Step 3: Divide into groups Objectives: ◦ Carefully select experimental and control groups ◦ Homogenize the two groups ◦ Isolate the treatment by controlling for potential extraneous variables Common flaws in design: ◦ Control group does not resemble treatment along important variables ◦ Experiment is not double blind ◦ Participants can guess the experiment, allowing a placebo effect to occur

Question 54

Good Experimental Design - Step 4 ◦ Objectives ◦ Common flaws in design

Answer 54

Step 4: Random Sampling Objectives: ◦ Make sure all members of the population are represented ◦ Ideally each member has as equal change of being selected ◦ Meeting these criteria is often not possible for practical reasons Common flaws in design: ◦ Sampling is not truly random ◦ Sample does not represent the population of interest

Question 55

Good Experimental Design - Step 5 ◦ Objectives ◦ Common flaws in design

Answer 55

Step 5: Random Assignment Objectives: ◦ Individuals who have been sampled are equally likely to be assigned to treatment or control ◦ Consider matching along potential extraneous variables which have been pre-selected Common flaws in design: ◦ Groups are not properly matched ◦ Assignment is not perfectly random

Question 56

Good Experimental Design - Step 6 ◦ Objectives ◦ Common flaws in design

Answer 56

Step 6: Measurement Objectives: ◦ Make sure measurements are standardized ◦ Make sure instruments are reliable Common flaws in design: ◦ Tools are not precise enough to pick up a result ◦ Instruments used for measurment are not reliable

Question 57

Good Experimental Design - Step 7 ◦ Objectives ◦ Common flaws in design

Answer 57

Step 7: Test the hypothesis Objectives: ◦ Use statistics to check for a significant difference ◦ Assign a pre-established threshold at which the null hypothesis will be rejected Common flaws in design: ◦ Small sample size leads to insufficient power ◦ Researchers do not set thresholds in advance and make afer-the-fact conclusions that lead to logical fallacies

Question 58

How does impression management threaten internal validity?

Answer 58

◦ Participants adapt their responses based on scoail norms or perceived researcher expectations ◦ Self-fulfilling prophecy ◦ Methodology is not double-blind, Hawthorne Effect

Question 59

How does a lack of reliability threaten internal validity?

Answer 59

Measurement tools do not measure what they porport to - lack consistency

Question 60

How can demand characteristics be a threat to internal validity?

Answer 60

Participants interpret what the experiment is about and subconsciously respond in ways that are consistent with the hypothesis

Question 61

What does it mean if an experiment doesn't reflect the real world?

Answer 61

◦ The laboratory setup doesn't/won't translate to the real world ◦ There is a lack of generalizability ◦ Is a threat to external validity

Question 62

How can selection criteria threaten external validity?

Answer 62

◦ If too restrictive of inclusion/exclusion criteria for participants (ex. sample is not representative)

Question 63

How can situational effects threaten external validity?

Answer 63

◦ Presence of laboratory conditions changes outcome (ex. pre-test and post-test, presence of experimenter, etc.)

Question 64

How can lack of statistical power threaten external validity?

Answer 64

◦ If sample groups have high variability ◦ If sample size is too small

Question 65

What are the four threats to external validity?

Answer 65

1. Experiment doesn't reflect real world 2. Selection criteria 3. Situational effects 4. Lack of statistuical power

Question 66

What are the six threats to internal validity?

Answer 66

1. Impression management 2. Confounding variables 3. Lack of reliability 4. Sampling bias 5. Attrition effects 6. Demand characteristics

Question 67

Define external validity

Answer 67

The extent to which the findings of a study can be generalized or applied to other situations, people, settings, or measures - it determines if the results of the study are applicable beyond the specific context in which it was conducted ◦ A lack of external validity might make it difficult to apply the studies conclusions to the real world

Question 68

Define internal validity

Answer 68

◦ Assess the extent to which a study's results accurately reflect a causal relationship b/w variables, without the influence of other factors ◦ A lack of internal validity can means an inherent flaw in the design of a study

Question 69

When is internal validity high?

Answer 69

When confounding variables have been considered and minimized and the causal relationships b/w the independent and dependent variable can be established by the way the experiment was set up

Question 70

Define demand characteristics

Answer 70

The tendency of participants to consciously or subconsciously act in ways that match how they are expected to behave ◦ They can threaten internal validity

Question 71

Define predictive validity

Answer 71

The extent to which a test or measurement predicts future outcomes or behaviours ◦ Needs to be considered especially in psychometric evaluations

Question 72

When do ethical problems arise most frequently?

Answer 72

In experimental designs b/c researchers are directly manipulating variables, not just observing what they see in nature

Question 73

What must be done to sure that ethical standards are met (in an experiment)?

Answer 73

◦ Modern experiments must be cleared by an independent interal commission ◦ The participants must be given a disclosure ◦ The experimental protocol should include debriefing ◦ In some cases, participants may be offered access to treatment or counseling services as part of debriefing (if the experiment may have triggered psychological vulnerability)

Question 74

Define disclosure

Answer 74

An outline given to study participants before the experiment begins that clarifies incenteives and expectations while reminding them of their right to terminate the experiment at any time

Question 75

Define debriefing

Answer 75

Where participants are told after an experiment exactly what was done and why the experiment was conducted

Question 76

What are the 8 types of non-experimental study designs?

Answer 76

1. Correlational studies 2. Ethnographic studies 3. Twin studies 4. Longitudinal studies 5. Case studies 6. Phenomenological studies 7. Surveys 8. Other studies: ie. Archival studies, Biographical studies, Observational studies, etc

Question 77

When might a non-experimal design be used?

Answer 77

When experiments are not feasible for practical or ethical reasons

Question 78

What is the benefit of a study with a non-experimental design? What is the trade-off?

Answer 78

They tend to offer the benefit of observing phenomena in a more naturalistic setting, often improving external validity - but with the trade-off of reduced control of the variables of interest, which tends to reduce the internal validity

Question 79

What is a correlational study? Their advantage and disadvantage?

Answer 79

◦ Measures the quantitative relationship b/w two variables ◦ Most commonly, Pearson correlation ◦ Advantage: showing a numerical relationship b/w two variables and are easier to conduct than experiments (b/c measurements on a population can be taken and variables do not need to be directly manipulated) - its a good preliminary technique ◦ Disadvantages: Causality cannot be infered and it may not detect a nonlinear relationship

Question 80

What is an ethnographic study? Advantages and disadvantages?

Answer 80

◦ Deep, lengthy qualitative analysis of a culture and its characteristics ◦ Advantages: provides detailed analysis and comprehensive evaluation ◦ Disadvantages: researcher's presence may affect individuals' behaviour and it is heavily dependent on the researcher conducting the study, is difficult to replicate and objectivity may be compromised

Question 81

What is a twin study? Advantages and disadvantages?

Answer 81

◦ Analysis of heritabuility through measuring characteristics of twins ◦ Advantages: offers insight into how nature and nurture might interact to lead to various characteristics ◦ Disadvantages: it is difficult to find participants who meet criteria and it is difficult to analyze the complex variables involved and how they interact

Question 82

What is a longitudinal study? Advantages and disadvantages?

Answer 82

◦ Long-term analysius that intermittently measure the evolution of some behaviour or characteristic ◦ Advantage: scientists can understand how trait of interest changes over time ◦ Disadvantage: logistically demanding, expensive, difficult to implement and has a high attrition rate

Question 83

What are case studies? Advantages and disadvantages?

Answer 83

◦ Deep analysis of a single case or example ◦ Advantage: offers comprehensive details about the single case ◦ Disadvantage: results may not be generalizable and does not offer points of reference or comparison

Question 84

What are phenomenological studies? Advantages and disadvantage?

Answer 84

◦ Self-observatio of a phenomenon by researcher or small group of participants ◦ Advantages: Introspection can provide insight into behaviours and occurences that are difficult to measure ◦ Disadvantages: lacks objectivity due to results coming from self-analysis (and small to 1, sample size), and it can be difficult to generalize results to other circumstances or individuals (reduces external validity)

Question 85

How is a survey used for studies? Advantages and disadvantages?

Answer 85

◦ Use of a series of questions to allow participants to self-report behaviours or tendencies ◦ Advantages: it is easy to administer and it can provide quantitative data that can be compared to large participant pools ◦ Disadvantages: self-reporting creates limitations in objectivity

Question 86

What are archival studies? Advantages and disadvantages?

Answer 86

◦ Analysis of historical records for insight into a phenomenon ◦ Advantages: provides insight into events from the past that are unique from everyday behaviour ◦ Disadvantages: quality of analysis subject to the quality and integrity of records, difficult to conduct follow-ups, and data is unlikely to be comprehensive, leaving ambiguity, and unanswered questions

Question 87

What are biographical studies? Advantages and disadvantages?

Answer 87

◦ Exploration of all the events and circumstances of an individual's life ◦ Advantages: Comprehensive knowledge of all the details of an individual's life ◦ Disadvantages: limitations in objectivity and difficult to generalize observations

Question 88

What are observational studies? Advantages and disadvantages?

Answer 88

◦ Broad category that includes any research in which experimenters do not manipulate the situation or results ◦ Advantages: naturalistic observation of circumstances as they are ◦ Disadvantages: difficult to tease out the complex interplay of many variables

Question 90

Describe how Pearson correlation works

Answer 90

◦ Assigns a number from -1 to +1 to a pair of variables ◦ If the value is negative, the two variables are negatively correlated (if one increases, the other will decrease) ◦ If the value is positive, the two variables are positively correlated (if one increase, so does the other) ◦ A value of zero increases no correlation and that there is a linear relationship b/w the two variables (although a non-linear relationship is still possible) *Remember that correlation doesn't mean causation ◦ Significance testing can be combined w/Pearson correlations to see if the determined correlation is likely to have occured by chance or not

Question 91

Define heritability

Answer 91

The extend to which an observed trait is due to genetics vs. the environment

Question 92

What is a cross-sectional study?

Answer 92

◦ Similar to a longitudinal study, but with data collection or surveys of a population or sample at a specific time