chapter 11

Question 1

what is the problem in analyzing experimental data

Answer 1

If the independent variable has an effect on the dependent variable, the means for the experimental conditions should differ

Total variance = systematic variance + error variance

However, error variance can also cause the means to differ
- So, the condition means could be different even off the independent variable had no effect

Then, how do we know whether the difference among means is caused by the independent variable or by error variance?

Question 2

sources of error variance

Answer 2

individual differences
transient states
environmental factors
differential treatment
measurement error

Question 3

individual differences

Answer 3

pre-existing differences between people; this is the most common source of error variance

Question 4

transient states

Answer 4

at the time of the experiment, participants differ in how they feel (e.g., mood, health, fatigue, interest)

Question 5

environmental factors

Answer 5

differences in the environment in which the study is conducted (e.g., noise, time of day, temperature)

Question 6

differential treatment

Answer 6

despite their best efforts, experimenters do not always treat all participants exactly the same

Question 7

measurement error

Answer 7

unreliable measures increase error variance

Question 8

3 solutions to understand whether the difference is meaningful

Answer 8

significance testing
effect size
confidence intervals

Question 9

significance testing

Answer 9

the probability that the difference between the groups is due to error variance

Estimate how much the means should differ if the independent variable has no effect

If the observed mean difference exceeds this amount, then the independent variable may be having an effect

We cannot be certain that the difference was caused by the independent variable, but we can know the probability that the independent variable caused the means to differ

Question 10

effect size

Answer 10

the size of the difference between the groups is noteworthy or not

Question 11

confidence intervals

Answer 11

the difference between the groups relative to the precision of the data

Question 12

null hypothesis statistical testing

Answer 12

used to determine whether differences between the means of the experimental conditions are greater than expected on the basis of error variance alone

Question 13

null hypothesis

Answer 13

independent variable did not have an effect on the dependent variable

Question 14

experimental hypothesis

Answer 14

the independent variable did have an effect on the dependent variable

Although we are really interested in the experimental hypothesis, inferential statistics test the null hypothesis

Question 15

problems with null hypothesis

Answer 15

p-hacking

information it provides is not as precise and informative as other approaches

Question 16

p-hacking

Answer 16

p-value fishing

when researchers over analyze their data in search of significance (needed for publication)

Performing many unplanned analyzes increases the likelihood of finding effects of the basis of type I error alone

Question 17

p-value

Answer 17

the probability that the obtained difference between the condition means is due to error variance

Ranges from .0 to 1.

The closer you get to 1, the more likely that the difference between the means is exactly what one could expect based on the amount of error variance

• A p-value of 0.05 means the probability of getting our difference on the basis of error variance is 0.05 or 5%
• A p-value of 0.01 means that the probability of getting our difference on the basis of error variance is .01 or 1 in a 100
• A p-value of 0.001 means the probability of getting our difference on the basis of error variance is only 1 in 1000

Question 18

Rejecting the null hypothesis

Answer 18

researcher concludes that the null hypothesis is wrong and that the independent variable did have an effect (there is a group difference!)

Question 19

Failing to reject the null hypothesis

Answer 19

researcher concludes that the null hypothesis is correct and that the independent variable did not have an effect (there is no group difference!)

We cannot say that we “accept” the null hypothesis, because the null hypothesis can never be proven as there may be several other explanations

Question 20

Statistically significant

Answer 20

has low probability of occurring as a result of error variance alone; when we reject the null hypothesis with a low probability of making a Type I error - the difference

Question 21

type I error

Answer 21

a researcher rejects the null hypothesis when it is true (i.e., showing sig finding when in fact there was no real difference between groups)

alpha
false positive

example:
- finding a drug is effective when it is in fact is not
- sending an innocent person to jail

Question 22

alpha

Answer 22

the probability of making a Type I error (and erroneously believing that an effect was due to IV when it was actually due to error variance)

Question 23

false positive

Answer 23

test shows positive when in fact there is no virus

Question 24

type II error

Answer 24

a researcher fails to reject the null hypothesis when it is false

beta
false negative

example:
- failing to find that a drug works when in fact it does
- failing to send a guilty person to jail

Question 25

type I vs type II error

Answer 25

You have a choice! --> You can either set Alpha (type I error) low or Beta (type II error) low ---> But when you set Alpha low, you will increase your type II error --> Set the probability of committing type I error (Alpha level) low -- Pick Alpha 0.001 instead of 0.05 -- Probability of committing type II error goes up

Question 26

typically in behavioral science (alpha)

Answer 26

We set the Alpha level at 0.05 or 0.01 or 0.001 We set Beta level (probability of committing type II error) 0.80 (i.e., 80%)

Question 27

power

Answer 27

probability that a study will reject the null hypothesis when it is false; probability that a test will reject the null hypothesis when the alternative hypothesis is true Ability of your design to show that a drug is effective, when in fact it is Power = 1-Beta Beta == type II error Power increases with the sample size

Question 28

power analysis

Answer 28

used to decide how many participants are needed to detect a significant effect

Question 29

effect size

Answer 29

magnitude of effect of IV on DV or the strength of the relation between two variables

Question 30

effect size helps determine real-world relevance:

Answer 30

An effect can be statistically significant but too small to matter practically An effect can be non-significant but potentially impactful if the study is underpowered

Question 31

effect size indicators

Answer 31

r-squared or eta-squared or omega-squared cohen's d odds ration (OR)

Question 32

r-squared or eta-squared or omega-squared

Answer 32

the proportion of variance in the dependent variable accounted for by the independent variable

Question 33

cohen's d

Answer 33

the difference between the two means relative to the size of the standard deviation of the scores

Question 34

odds ratio (OR)

Answer 34

the odds of an event occurring in one group relative to another group = 1 → if the event is equally likely in both groups, the odds ratio is 1 > 1 → greater than 1 shows that the odds of the event are greater in one group than in another < 1 → less than 1 indicates that the odds in one group are less than in the other (0.5 means half the odds)

Question 35

effect size and significance

Answer 35

magnitude vs significance? - A p-value indicates whether an observed result is statistically significant - Effect size indicates the strength of the observed phenomenon

Question 36

ex study 1: p<0.01, Cohen's D=0.2

Answer 36

large sample, small effect significant effect although the effect is statistically significant due to a large sample size, the small effect suggests the drug's practical impact is minor

Question 37

ex study 2: p=0.07, cohen's D=0.9

Answer 37

small sample, large effect although the p-value isn't below the significance threshold, the large effect size indicates the drug could have a real impact the non-significance may be due to the N, and increasing the sample size could make this effect significant

Question 38

effect size and power

Answer 38

Power depends on the effect size: larger effect sizes increase the power, making it easier to detect true differences or associations When planning a study, researchers use the expected effect size to calculate the necessary sample size to achieve sufficient power (often 80% or higher) - If the effect size is small, you will need larger sample size to boost the power - If the effect size is large, you may need a smaller sample to have enough power to detect the effect you are looking for

Question 39

confidence intervals

Answer 39

range of values around the point estimate in which the population value is most likely to fall Gives us a sense of how much faith we can have in the statistics we calculate in a study

Question 40

How do we interpret the confidence interval for a sample mean?

Answer 40

A range of values in which the population mean likely falls

Question 41

Confidence interval gives:

Answer 41

point estimate interval estimate

Question 42

point estimate

Answer 42

its most likely value based on what we know from our sample data

Question 43

interval estimate

Answer 43

range of values around the point estimate in which the population value is most likely to fall

Question 44

relationships (power; effect size; sample; CI)

Answer 44

Effect size → power will increase (easier to detect real effect) Sample size → power will increase; CI narrows (more precise); less Type II error (easier to detect effect) Power → increased likelihood of detecting real effect (1-Beta); reduces type II CI width decreases → estimate precision increases; significance testing becomes more reliable