Lecture 6 - Evaluating Research Flashcards
(24 cards)
Conceptualisation
- Process of defining a specific concept
- Aims to refine and specify an abstract concept
Operationalisation
- Process by which a research specifies how a
concept will be measured - Define specific definition that will bring about empirical observations representing the abstract concept in the real world
Measurement Error
The difference between the observed value and the actual / true value
Random Error
Creates imprecision in data and impacts how reproducible the result would be under equivalent conditions. Causes:
- Fluctuations in the environment
- Variations in respondents/researchers’ mood
- Respondent misread a question
Systematic Error
Creates bias in data and means that measurements of the same thing will vary in predictable ways. Causes:
- Response bias / Social desirability bias
- Sampling bias
- Improper instrument
Reliability
Refers to the accuracy and consistency of data collection were it to be conducted in more than once place and/or at more than one time.
Test-retest Reliability
Types of Reliability
Assesses the extent to which the score on a measured variable is consistent on multiple occasions (across time).
- Example: Respondents take an IQ test in January and then, to reaffirm the results they get to take the test again in March. If they get the same / similar results, this indicates high test reliability
(Equivalence) Internal Consistency
Types of Reliability
Assesses whether you get the same results when a specific concept is assessed via multiple items (survey questions)
(Equivalence) Inter-subjective
Types of Reliability
Refers to the degree to which multiple observers agree on measurement
- Example: Three different lecturers grade the same student’s paper using the rubric and give substantially different grades. This indicates that there is low inter-rater reliability (e.g. because the rubric criteria are too vague).
Validity
Refers to the adequacy or relevance of data collection in measuring what it is supposed to measure
- E.g. Validity is low if the research design is not appropriate to the research question
Content Validity
Type of Measurement Validity
Does the measure of adequacy capture the concept’s full meaning? Conceptualisation
Criterion Validity
Type of Measurement Validity
Measures consistency between different measures for the same concept? Operationalisation
Construct Validity
Type of Measurement Validity
Shows the relationship between measures for different concepts.
- Strong correlation between the concepts = convergent validity
- Weak correlation between the concepts = **discriminant validity **
Internal Validity
Type of Design Validity
- Concerned with Cause-and-Effect relationship. Does the IV ‘’cause’’ the DV?
- Can only be truly tested in experiments (or survey experiments)
- Considerations:
1. Temporal Precedence
2. Correlation between the two variables
3. Rule out plausible alternative explanations (controls) - Threats:
1. History
2. Maturation and attrition
3. Repeatedly retaking a test (knowledge vs. memory)
4. Sampling bias (between groups)
Two Types of External Validity
- Population Validity
- Ecological Validity
Population Validity
Type of External Validity
Can the findings be generalised to the
population?
Ecological Validity
Type of External Validity
Can the findings be generalised across
settings / situations?
Coding
“Coding is the process of analyzing qualitative text by taking them apart to see what they yield before putting the data back [organising] in a meaningful way.”
- What can you code? Documents, transcripts, videos, audio, images
- What can you code for? Depends on what you are looking for. e.g. content, language (discourse), them
Credibility
Trustworthiness
Establishing that the results are
credible or believable.
How?
- Respondent validation
- Triangulation (multiple data sources/methods
Transferability
Trustworthiness
Applicability of research
findings to other settings (generalising).
How?
- Offering thick descriptions
- Saturation (e.g. theoretical, data, coding)
Dependability
Trustworthiness
The researcher accounts for the
ever-changing context within which the research occurs.
How?
- Keeping an audit trail.
- Transparent codebook & Inter-coder reliability
Confirmability
Trustworthiness
To what extent can the results
be confirmed/corroborated by other
researchers? (replicability).
How?
- Keeping an audit trail.
- Transparent codebook & Inter-coder reliability
Reflecivity
Trustworthiness
The researcher examines their own
positionality and makes them known.
How?
- Ethical discussions
- Discussion on positionality
Temporal Precendence
Te principle that the cause must happen before the effect in order to establish a causal relationship in research.
