Validity, Measurement, Reliability, Descriptive Research

Question 1

Independent variable

Answer 1

manipulated variable (e.g. condition / congruence of colors and words presented)

Question 2

dependent variable

Answer 2

measured variable (e.g. time needed to read out words)

Question 3

control variable

Answer 3

variable that is being held constant on purpose (e.g. number of words)

Question 4

confounding variable

Answer 4

variable that change simultaneously with the independent variable (e.g. time point of testing, order of words)

Question 5

random error

Answer 5

noise during testing

Question 6

conceptual variable

Answer 6

syn: construct
= variable stated at an abstract (theoretical) level (e.g. anxiety is to be measured)

Question 7

operational variable

Answer 7

= operational definition
= specific way in which a construct is manipulated / measured in a study (e.g. test score for anxiety)

Question 8

What are the possible claims in experimentation?

Answer 8

1. Frequency claims: focuses on 1 variable -> descriptive research
2. Association claims: focuses on relationship between at least 2 variables
3. Causal claims: focuses on causation, so a change in 1 variable is responsible for changing the value of another variable -> experimental research

Question 9

Validity

Answer 9

way of knowing whether the claims are good
= appropriateness of a claim

Question 10

complete the sentence: a valid claim is…

Answer 10

1. reasonable
2. accurate
3. justifiable

Question 11

Construct validity - definition + question to be asked

Answer 11

= an indication of how well a conceptual variable is measured / manipulated in the study
Question to be asked: “How well is a construct operationalized?”

Question 12

Threats to construct validity

Answer 12

• inadequate operational definitoin (e.g. does test really measure anxiety or just the absence of self-confidence?)
• mono-operation bias

Question 13

mono-operation bias

Answer 13

Mono-operation bias occurs when a single measure or a single method is used to assess a complex theoretical construct.

Question 14

External validity - definition + question to be asked

Answer 14

= indication of how well the results of a study generalize to, or represent, individuals, settings, places, and times (contexts) besides those in the study itself
Question to be asked: “Is it possible to generalize?”

Question 15

Threats to external validity

Answer 15

• selection biases
• study setting is different from other settings
• specific time at which the study is performed is different from other times

Question 16

How do you get external validity?

Answer 16

1. assigning participants to experimental and control conditions by chance
2. minimizing pre-existing differences between those assigned to the different groups

Question 17

Internal validity - definition + question to be asked

Answer 17

= the extent to which it is possible to rule out alternative explanations for a causal relationship between two variables
Question to be asked: “Can we rule out alternative explanations?”

Question 18

Threats to internal validity

Answer 18

• maturation
• history
• testing
• instrumentation
• regression to the mean
• attrition / experimental mortality
• selection effects (e.g. Simpson paradoxon)
• design confounds = the presence of another variable that unintentionally varies systematically with the independent variable
• observer bias (single-/double-blind studies recommended!!)
• demand characteristics
• placebo effects

Question 19

maturation

Answer 19

Maturation refers to changes that occur naturally over time within the participants of a study. These changes may affect the dependent variable independently of the independent variable. For example, if you are studying the effects of an educational intervention on children’s reading skills, normal developmental changes in reading ability that occur with age could be a threat to internal validity.

Question 20

regression to the mean

Answer 20

Regression to the mean refers to the tendency for extreme scores on a variable to move closer to the average (mean) when measured again. If participants are selected based on extreme scores, it may appear that a treatment had an effect when it’s merely a statistical artifact.

Question 21

attrition

Answer 21

Attrition occurs when participants drop out of a study before it is completed. If the reasons for dropout are related to the independent variable, it can bias the results. Experimental mortality is a similar concept, referring to participants who do not complete the study.

Question 22

demand characteristics

Answer 22

Demand characteristics refer to cues or hints that participants pick up during a study, which can lead them to guess what the researcher expects or wants. This can influence their behavior or responses in a way that doesn’t reflect their true reactions.

Question 23

Statistical validity - definition + question to be asked

Answer 23

= the extent to which a study’s statistical conclusions are accurate and reasonable
Question to be asked: “How well do the numbers support the claims?”

Question 24

Threats to statistical validity

Answer 24

• violated assumptions of the test statistics
• fishing and the error rate problem (seeing things that aren’t there)
• low statistical power (missing the needle in the haystack, unreliability of measures)

Question 25

measurement

Answer 25

a systematic way to assign numbers or names to objects and their features

Question 26

measurement scales

Answer 26

1. nominal scale -> just categories 2. ordinal scale -> ranked categories 3. interval scale -> adding / subtracting possible 4. ratio scale -> the absolute is zero, therefore multiplying and dividing possible

Question 27

how do we calculate reliability

Answer 27

observed score = true score + error score -> observed score = measured value -> true score = real value -> error score = difference between measured and real value

Question 28

types of reliability

Answer 28

1. test reliability 2. interrater reliability 3. experimental reliability

Question 29

reliability

Answer 29

Reliability refers to the consistency and trustworthiness of a system, product, or process in delivering consistent and dependable results over time.

Question 30

Test reliability

Answer 30

Test reliability is the extent to which a particular assessment or measurement instrument produces consistent and stable results when administered to the same individuals on different occasions.

Question 31

Interrater reliability

Answer 31

Interrater reliability is the degree of agreement or consistency between different raters or observers when assessing or scoring the same data, typically used to assess the reliability of subjective judgments or evaluations.

Question 32

Experimental reliability

Answer 32

Experimental reliability pertains to the consistency and dependability of results obtained in a scientific experiment, ensuring that the same experiment conducted under similar conditions would yield similar outcomes.

Question 33

Types of test reliability

Answer 33

1. test-retest reliability: same results at time points 1 and 2 2. internal reliability

Question 34

Types of internal reliability

Answer 34

1. Split-half reliability 2. Cronbach's alpha

Question 35

Cronbach's alpha

Answer 35

Cronbach's alpha, often referred to simply as "alpha," is a statistical measure used to assess the internal consistency or reliability of a set of items or questions in a research survey or test. It is named after its developer, Lee Cronbach, and is widely used in fields such as psychology, education, and social sciences to evaluate the consistency of responses within a questionnaire or scale. The primary purpose of Cronbach's alpha is to determine whether the items in a survey or test are measuring the same underlying construct or concept. In other words, it helps researchers assess whether the items are reliably measuring what they are intended to measure. Cronbach's alpha produces a score between 0 and 1, with higher values indicating greater internal consistency.

Question 36

methods of descriptive research

Answer 36

observations, surveys, case studies

Question 37

problems of descriptive research

Answer 37

- people are not always good at observations - low internal validity - threats to construct validity: observer bias, observer effects, reactivity, strategy, "wait it out" attitude, unobtrusive observations / measures

Question 38

problems of surveys

Answer 38

- low internal validity - threats to construct validity: question wording, question order, use of response sets, trying to look good / bad, incapability of reporting feelings

Question 39

problems of frequency claims (e.g. study shows that 3 in 4 women are sad when doing xyz)

Answer 39

- low internal validity - threats to external validity: biased sample, unrepresentativeness -> can be improved by random sample, replication, etc. - threats to statistical validity: samples too small

Question 40

what are important questions in descriptive research?

Answer 40

1. how was each variable measured? -> construct validity 2. how were the results generalized? -> external validity 3. how do the numbers support the conclusions? -> statistical validity