Final Flashcards
(102 cards)
1
Q
In ethics, what do you need to worry about?
A
- Test/ Questionnaire selection
- Participants recruitment
- Treatment of participants
- Scoring and interpretation
2
Q
What is part of the treatment of participants?
A
Even decisions made prior to seeing any participants are part of ethical measures
- Understand your measures
- Follow appropriate procedures
- Ensure participant wellbeing (don’t waste their time)
- Give back feedback and make sure you’re the right person to do so
3
Q
What is informed consent?
A
To make sure participants know what they’re agreeing to
Includes: estimating time commitment, their right to withdraw and goals of study but not hypotheses/ deception
4
Q
How do we ensure the confidentiality of participants?
A
By handling the data responsibly
5
Q
Should we deliver feedback appropriately after the experiment?
A
Yes, and should benefit participants
6
Q
What is part of responsible data handling?
A
- No unauthorized access to your data (even government, can give access to people that can prevent/ improve mistakes)
- Safe retention of data (anonimity+ keep 5 years before destroying)
- In testing, ensure data interpreted appropriately
7
Q
Is data removal allowed?
A
Yes, but need to be done responsibly
8
Q
What consists of responsible data removal?
A
- Balancing need to respect participant’s contribution with the need to research accurate conclusions
- Participants have the right to try to derail our study but not succeed at it
9
Q
What should we always remember in data removal?
A
- Our statistics have assumptions we should meet
* Many statistics require complete data sets
10
Q
What is the purpose of the APA guidelines?
A
To ensure usefulness and proper application of the techniques
In questionnaires: privacy, right people, correct setting, amount of time (pressure, back and forth, limited, etc.)= use tools correctly for tight purposes
11
Q
Give an example of when the responsible use of questionnaires should be used and why? (hint: Beck Depression Inventory)
A
- Each category assess one aspect of depression
- Need to make sure the questionnaire doesn’t cause harm to participant (in this case, might cause a negative mood)
- For suicidal thoughts and wishes, might create problems in terms of ethics, the role of the researcher to ensure participants don’t harm themselves (well-being of participants)
- Need to make sure researcher is ok to deal with potential fall back
12
Q
What do ethics applications must establish?
A
Scientific merit, safe handling of data and protection of participant’s well-being
13
Q
Can we give out the personal data of participants?
A
No!
It is confidential info of participants and it is often regulated by the law
Protects the client/patient/participant
For research purposes, can give collective data but not individual
14
Q
What are the assumptions of Factor Analysis?
A
- Based on General Factor Model, factor analysis shares some important assumptions with classical measurement models or congeneric model
- Errors are random and not correlated with latent variable
- Items are correlated with each other because they share a common latent variable
15
Q
What is a latent variable?
A
Unobserved influences on our measurements or constructs that we are trying to measure
16
Q
What is a factor?
A
Another way to refer to a latent variable and component but debate with maths
17
Q
Why would we use the term factor more than latent variable?
A
Main reason to use these terms instead of latent variable is to better acknowledge that the unobserved influence was derived empirically (looking into data where patterns are).
18
Q
What is factor analysis?
A
The process of trying to identify the latent variables that influence our measurements
19
Q
In factor analysis, what forms identifiable clusters?
A
Items that have a stronger association (correlations) with each other but weaker associations with other items will form identifiable clusters
20
Q
What are we capitalizing on in correlation across items?
A
Similarities and differences
21
Q
What happens if items correlate strongly?
A
There will be one factor/ latent variable identified
22
Q
What are the rules for good factor analysis?
A
-Need quite large data to work effectively (will probably allow us to recreate pattern later as well)
It needs to be larger than for internal consistency reliability or validity analyses
23
Q
What does a simple construct should produce?
A
One factor
24
Q
What are questionnaires meant to assess
A
One simple construct
25
What is an iterative process and what should be viewed as such?
Iterative process: going through a cycle by starting over with change made
Factor analysis
26
What is some general info about items in factor analysis?
- What you put in analysis dictates what you get out of it (garbage items=garbage analysis)
- Adding or dropping even one item will change the outcome (small or big shift)
- Every item has potential to create an item or influence creation of other factors
27
True or false: The more items you have, the more likely you are to find one specific factor
False!
The more items you add to a questionnaire, the less likely it becomes that you will find only 1 factor (so should remove extra questions to measure the 2 constructs)
28
What gets called factors?
The important clusters
29
How do we define the importance of a factor?
-Typically, when a factor has an eigenvalue bigger than 1
30
What does an eigenvalue meausre?
Measures the amount of information captured by a cluster/ pattern identified in the data
31
What does an eigenvalue of 1 mean?
Indicates the factor captures as much information as one typical good item
32
What does parallel analysis create?
- Creates random data with the same number of variables and observations as your data
- Creates correlation matrix where eigenvalues are calculated
33
Parallel Analysis: What happens when eigenvalues from random data are larger than the eigenvalues from your real data?
Know real data not correlated better than random data
34
Does factor loading always need to be positive?
No!
| Negatives do not damage the apparent accuracy of the exploratory factor analysis
35
What do we use when exploratory factor analysis is left with a subjective decision to make?
Use our judgement! Some tools at our disposal to help
36
What is a scree plot and what does it do?
It is a visual representation of the eigenvalues obtained in analysis
-Shows which factor are above the 1 value, important to look for the steep vertical line
37
What is the minimum factor loading?
-1
38
What does a rotation allow us to see?
The spread of variability among our factors
39
What are the 2 forms of rotation
Orthogonal and Oblique
40
What is an orthogonal rotation?
It maximizes the squared variance in the factor loadings
- Clusters are as different as possible and unrelated
- Rotation is clock-wise after returned to vertical orientation for analysis. Can potentially reframe how we see things
- It makes it easier to identify the differences between items in relation to their clustering
41
What is the underlying theory of orthogonal rotation?
2 unconnected variables (like the original model)
42
What rotation is associated with varimax rotation?
Orthogonal rotation
43
What rotation is associated with oblimim rotation
Oblique rotation
44
What is an oblique rotation?
Maintains some relationship between the factors-clusters are different but related things
- Used to minimize the squared loading covariance of the factors, while allowing them to be correlated
- Don't have to look much for eigenvalues anymore
45
What do we expect to be the force on our measurement?
Only latent variable, other influences probably just random error
46
What does PCA stand for?
Principal components analysis
47
In PCA, what does it mean when we say that we expect all variance to come from a common source among the items?
- Means there is no choice of extraction method (one way to identify cluster only) as these options under EFA are based on different ideas of the sources of variance
- Quite possible for some of the common variance to still be error; accordingly we still want to discard any factors that seem too trivial on the basis of their eigenvalues
- Everything is essentially meaningful until we look at eigenvalues
48
Is PCA considered factor analysis? Why?
No
| Because in general, don't produce meaningfully different results as they are still doing essentially the same thing
49
When should PCA be used?
If you want to maximize the amount of explained variance
50
If someone asked you to do a factor analysis what should you do?
An exploratory factor analysis
51
What does a confirmatory factor analysis allow for?
Allows complete control over which items theoretically go with which latent variable; with the right software, you could even specify expectations for how strong the relations should be. Tell the software exactly what we want
52
What are the steps in the CFA process?
1-Need for a prior theory and check that theory
2-Input the model in Jamovi
3-Examine the model fit
4-If good fit, report the findings. If not, need to revise our model or our theory
53
What is considered a good CFI and TLI as model statistics?
above .95 and everything below .9 is really bad
54
What is considered a good RMSEA?
Want it to be below .05
55
To be significant, what should be the p value of the factor loadings?
Below .05
56
What is a disadvantage of an EFAr?
Allows very little control over the factor structure; theory can be introduced only through rotation or forcing a set number of factors
57
What does a CFA tell us?
If we met our theoretical expectations
58
If we are missing some data, what do we need to do in our CFA?
Need to calculate the mean for our calculations
59
What is the Hierarchical Factor Analysis?
Model where you have one or more latent variable causing changes in one or more latent variable
o You have a primary latent variable that influences the other latent variables
o If common cause, will be highly correlated
60
Why is it relevant to measure more than one factor?
Because in questionnaires, there can be more than one factor that we want to measure
Ex:
o A test for a course may not just capture one construct (reliability, validity, factor analysis)
o An IQ test captures G (overall intelligence) but it also has multiple components within it so people will vary, making distinct units
o A personality test assesses many aspects of personality
61
What are the 5 factors of the 5-factor model?
* Openness
* Conscientiousness
* Extraversion
* Agreeableness
* Neuroticism
62
How do we know if each factor of personality is different from one another?
If the correlations are significant in big enough samples
63
How much variance does the G factor explain in personality?
60%
64
Do we have a hierarchal model in personality?
No, close but we still have 2 factors
65
What is the other theory than the Big 5 to measure personality?
Stability (alpha, N, C, A) and plasticity (beta, E&O) also influencing personality so General factor for personality and these 2 are big factors as well
66
How do we know which one makes the most sense whether we have a personality or not?
1. One overall factor must account for a “great” amount of variance
2. Any other factors should be more marginal
3. The OCEAN subscales must relate strongly to the Big One
67
What is the Big One in personality?
Behavioural characteristic of being emotionally stable, agreeable, conscientious, extraverted, and intellectually open individuals in comparison to neurotic, disagreeable, careless, introverted and close-minded persons
68
What is the Big two in personality?
Combination of high versus low stability and plasticity where stability is somewhat more important than plasticity
69
What is the meaning of the word standardized?
1. Statistical manipulation where we convert our measurements to Z-score units prior to using them in an analysis (as done for r). Unstandardized means original units
2. Means a measure has an expected set of procedures or that the items on a questionnaire have been previously established and we didn’t change them, that a measure has established normative information available, or some combination of all these. Unstandardized means one or more of these is missing.
70
Could emotional intelligence also measure the Big One?
Yes
71
What do we mean when we say that standardized measures are proprietary?
That we have to pay for them
72
What's the difference between a standardized and unstandardized measure?
Standardized: Controlled administration conditions, normative information available, interpretation guidelines available, instruction manual available, carefully vetted (?) and probably costly
Unstandardized: May not require controlled conditions, typically lack normative information, less interpretable for participants, mainly useful for research purposes, can have equivalent or better reliability and validity
73
Are standardized measures better? Why or why not? Use an example from class
No, it's not (ex: (free personality questionnaires are more reliable and efficient than the paid variety): Hamby, Taylor, Snowden & Peterson)
• Considered the possibility that setting a cost for something doesn’t necessarily mean it’s better than something free
• Argued that, for measuring personality, free tests are better: more reliable and efficient (validity not assessed= problem)
74
What are the problems we can encounter if we don't evaluate the psychometric properties of our tasks?
o Making less-than-optimal design decisions (e.g., power, task selected)
o Failure to find predicted results
o Incorrectly interpreting results
75
Which Greek philosopher would have said that the unexamined measure should not be measured?
Socrates
76
What is the AX-CTP?
A cognitive task used to assess individual differences in cognitive control ability
The ability to actively maintain and use goal-directed information to regulate behaviour
77
What are the 2 forms in the Dual Mechanisms of Control?
* Proactive: Control is implemented in advance, through maintenance of contextual information
* Reactive: Control is implemented after an important event occurs
* Cooper, Gonthier, Barch & Braver (2017) examined the psychometrics of this task (believed that you can find stable measurable differences in proactive or reactive in responding)
78
What are 3 main qualities to address cognitive tasks such as the AX-CTP?
1. Discrimination: Scores aren’t entirely random (e.g., high/low performance by some individuals)
2. Reliability: Scores are a precise estimate of one’s ability
3. Validity: Scores reflect the right cognitive faculty
79
Can Inter-group biases affect task performance in undesirable ways?
Yes
Ex:
• Neurotypical young adults are strongly biased toward proactive control, setting performance at ceiling for all but AY trials (problem with understanding the differences between those groups)
• Reliability suffers in this case, but as the task was designed for clinical diagnostic purposes
80
What does proper measurement need?
o Need good theories
o Need to capture all our dimensions well
o Need reliability
o Need convergent validity (ideally also discriminant validity)
81
In the CES-D, why is content validity one of the main problems? And what effect can it have?
o CES-D has 20 items and 1/3 of them do not appear in any other commonly used measures of depression (shows disagreement)
o What effect can this have? Measuring something else in different studies so inconsistencies and affects replicability.
o Researchers could also engage in p hacking, only considering one correlation rather than all of the others that did not match expectations
82
In the CES-D, why is reliability a problem?
• Inter-reliability for major depression diagnoses was .28 (presumably this is an r)
• Cronbach’s alpha has many limitations but in most cases is the only statistic ever used to demonstrate reliability
o At least 20% of studies don’t even report alpha
• Can have poor construct validity and high reliability
83
In the CES-D, why is validity a problem?
• Much worse when looking at validity or rather not looking at it
• Adding or removing items is a common practice without providing a clear reason for doing so
o Changing items isn’t necessarily a bad thing; the lack of justification is
• Many studies don’t try to demonstrate validity and many that do simply provide a citation to some other study
84
Why is critical thinking important?
* To solve problems, need to increase knowledge about the importance of good measurement practices and be more critical of studies that ignore them
* Alpha can’t tell you anything about validity when it’s high (is internal consistency best way to assess validity)
85
What's the difference between a test and a questionnaire?
Test: Items have right or wrong answers
Questionnaires: Do not (except with respect to the truth of the responses for that individual).
86
Why might internal consistency reliability not make much sense for tests?
Depends on how many factors we expect to capture using the test, and how strongly correlated the factors are
87
What is discrimination?
Ability to accurately separate individuals into high/low performers
88
What is difficulty?
Likelihood of getting a question correct
89
What is the formula for discrimination?
Discrimination= (Ph/Nh)-(Pl/Nl)
Ph=People responding correctly (High performing group)
Pl=People responding correctly (Low performing group)
Nh=Number of people in group (H)
Nl=Number of people in group
90
What are the minimum and maximum discrimination scores?
Min: -1 (really bad)
Max: 1 (really good)
0=bad too
91
What is the goal of discrimination?
Goal of discrimination is good categorization of individual cases into high and low performers
o Means we ideally have approximately half of people getting items wrong and we want to be able to predict in which half an individual falls with relatively good accuracy- for each item
o So, there is typically a connection between discrimination ability and item difficulty
92
What is the formula for difficulty?
Difficulty= 1-(Correct/N)
93
What are the minimum and maximum discrimination scores? What's the ideal?
Min=0, Max=1, Ideal=0.5
94
In designing a test, should I want to maximize the variability in the scores (full range of possible responses)?
Yes!
| Means you should aim for most items to have a difficulty score near the optimal .50
95
With optimal difficulty, do you optimize the probability of getting a high discrimination score?
Yes but it is not certain
| Means your item can separate good performers from bad performers
96
What psychological considerations should be taken into account for difficulty?
People inclined to want to do well so might or might not be a concern
Sometimes, need to sacrifice some psychometric quality to accommodate psychological considerations
97
What can help us resolve the contrast between good psychometrics and psychological considerations? If you can, explain how it works.
Standardization!
• In this case, standardization means taking the raw scores which are likely very low and converting them into a new score that looks more acceptable
o Percentiles would be one way of doing it
o IQ scores typically use standardization via Z scores, where a Z of 0 becomes 100 (a psychologically pleasing number) and each Z difference of 1 adds or subtracts 15 points.
98
What is mindfulness and what scale measures it?
Mindfulness is an active open attention to the present moment, also involving non-judgmental thinking about our experiences and a clear awareness of what is happening at any moment
The Measure: The Mindful Attention Awareness Scale (MAAS)
99
What is the core construct of mindfulness?
State of consciousness
| Attention can vary from person-to-person
100
What steps did Brown and Ryan take to make a good scale/questionnaire to test mindfulness?
1-Started with large pool of questions
2-Experts reviewed and dropped some questions not related to constructs (calmness, wellbeing and patience)
3-Removed items with specific vocabulary and refined scale so that they could apply to general population
4-Shorter list reviewed by experts again
5-Several studies conducted to test scale
6-Assessed the validity of their questionnaire which showed:
• Positive correlations with openness to experience, emotional intelligence, and wellbeing
• Negative correlations with rumination and social anxiety
• A lack of association with self-monitoring
101
In the end, what's the problem with the MAAS scale?
Measured only one facet out of the 5 of mindfulness which is lack of attention.
If want to measure that aspect, it is a good scale otherwise not really
102
Which scale was created to measure mindfulness taking inspiration from the MAAS scale?
The FFMQ measures the 5 factors of mindfulness: nonreactivity, observing, awareness, describing, nonjudgement
