Class 9: Good research design Flashcards
(26 cards)
Qiu et al.: What do they set out to study regarding social media and job market papers?
Whether social media promotion has a causal impact on job market outcomes and whether it can be used to address under-representation in academic
Qiu et al.: Research methods incl. IV and DV
Field experiment during 2022-23 economics job market in US and Europe
Job market candidates create post about their paper which is shared on a dedicated Twitter account
IV: Quote-tweeting by a prominent economist working in the same field as the candidate -> stratified randomization with under-represented groups assigned to treatment with 2/3 probability
DV: Number of views and likes for original tweet + job market outcomes like interviews, flyouts, job offers, salarry and satisfaction with job placement
Qiu et al.: What are their 3 main findings?
Posts assigned to be quote-tweeted receive 442% more views and 303% more likes
Being quote-tweeted had a positive effect on job market outcomes -> receiving one more flyout on average (but not interviews, salary or job satisfaction)
Positive effect for women, with women in treatment group reciving 0.9 more job offers
Qiu et al.: What are the 3 potential mechanism that can explain the results, and what do they find evidence for?
Attention/visibility mechanism: candidates with high-reach influencers get more views and likes, but job market outcomes better when influencer posts less frequently, indicating focused academic attention
Endorsement: endorsement level of quote-tweet did not affect outcomes, but academic reputation of the influencer and the JMC did affect job market outcomes positively
Candicate confidence: measured as tweet volume before and after treatment to see if they gain confidence -> but no difference found
Qiu et al.: What do they use their results to conclude?
That social media is a low-cost tool to increase visibility and improve job prospects for underrepresented groups, esp. women in economics
Clarke et al.: What is methodological incongruence?
Peer reviewer and editor comments that do not align with the authors’ research values and methdology
Clarke et al.: What often contributes to methodological incongruence?
Universalizing quantitative or qualitative values, concepts, and practices
Clarke et al.: Research methods
Qualitative survey asking qualitative researchers about the methodologically incoherent comments they have received from peer reviewers and editors, and how they addressed them - 2022, with 163 participants
Clarke et al.: What 4 categories of methodologically incoherent comments did the authors find?
Inappropriate universalization
Strategies for navigating methodologically incoherent comments
Power dynamics, loss, and (emotional) labor
Recommendations for improving integrity of peer reviews
Clarke et al.: What is inappropriate universalization?
Applying standards from quantitative (or specific qualitative traditions) as universal markers of good research across all qualitative methods, even when not appropriate - for example, sample size, quantification, hypotheses, generalizability, etc.
Clarke et al.: What are 3 strategies for navigating methodologically incongruent feedback?
- Educating -> explaining why changes won’t be made
- Preemptively explaining in article why certain practices were not used
- Caving, compromising or submitting elsewhere
Clarke et al.: What did power dynamics, loss, and (emotional) labor involve? (3)
Authors feel powerless and their research situated as less-than quantitative research
Subtle racism and intellectual superiority regarding Global North/South
Feeling tired and frustrated, questioning continuing with research
Clarke et al.: What are 3 recommendations for improving peer review?
- Journals must state which kinds of qualitative research is within their scope and ensure that the editor and reviewer has the right expertise
- Editors must find suitable reviewers and intervene when peer reviewers make incongruent comments and educate them
- Reviewers should not accept invitation to review if they do not have the appropriate methodological knowledge
Clarke et al.: Why does methodological incongruence matter?
Because while peer review is tended to make research better, incongruent comments make it worse
Huntington-Klein: What is empirical research?
Research that uses structured observations from the real world to attempt to answer questions
Huntington-Klein: A bad research design leads to…
Inconsistent results
Huntington-Klein: A research question must be these 2
Answerable
Improve our understanding of the world, i.e., inform theory
Huntington-Klein: What is data mining and what is it good/bad for?
Looking at patterns in the data to derive a RQ
Good for determining patterns and making predictions under stability
Bad for improving our understanding (why do these patterns exist) and tends to generate false positives
Huntington-Klein: What are the 5 different types of variables?
Continuous: Can take any numerical value
Count: Count something, for example number of businesses in France
Ordinal: Some variables are “more” than others, but no rule as to how much more nor equal space between
Categorical: Which category an observation is in - no order
Qualitative: Catch-all category for everything else, for example a specific news headline
Huntington-Klein: What is a distribution?
Description of how often different values of a variable occur
Huntington-Klein: What is a mean, percentile, median, range?
Mean: Adding all values, diving by number of values. Describes the average
Percentile: The Xth percentile is the value for which X% of the observations are less -> 5th percentile is the 5th value out of 100
Median: 50th percentile, representing a typical observation
Range: difference between min and max values
Huntington-Klein: What is variance, standard deviation, interquartile range, and skew?
Variance: measures variation of the data -> the bigger the variance, there more variation in the data
Standard deviation: How spread out the values are -> if small, the values are closer to the average, if large the values are more spread out
Interquartile range: Difference between 75th and 25th percentile -> covers half of your sample closest to the median
Skew: Whether the distribution leans to one side or another -> right skew if many observations to the right
Huntington-Klein: What is the theoretical distribution?
The distribution of all the data - even the data you didn’t collect
Huntington-Klein: What are two forms of distributions?
Normal distribution: Symmetric, used for aggregate values
Log-normal distribution: Has a heavy right skew, but once the logarithm is taken, it turns into a normal distribution -> useful for income, wealth etc.
