module 12 Meta-analysis Flashcards
(13 cards)
Meta-analysis
In this module, you will be introduced to meta-analysis. Meta-analysis is a formal approach to combining and systematically assessing previous research results that have investigated the same outcomes to derive conclusions about that body of research.
It is a powerful approach, and the advantages of meta-analyses include improved precision, the ability to answer questions not posed by individual studies, and the opportunity to settle controversies arising from conflicting findings.
STEPS IN META-ANALYSIS;
Step 1: Define variables; Need to be very clear what is included and what is not. Be clear about concept- how broad or narrow is it. Be clear about measurements.
Step 2: Literature search;search many databases, search conferences, reviews, dissertations, references etc etc.Search with multiple terms capture all eg depression/psychopathology/well-being etc etc. Do not be limited by english language only.
Step 3: Data collection;collect data/analysis results from the lit review,the data analysed may be in many forms so may need to work out other bits eg effect sizes etc,work out descriptives that you can then calculate and analyse yourself if req,some things may not have been analysed fully so may need to do this.May need to transform things so all can be similarly compared etc.
Step 4: Data entry;In the context of meta-analysis, data entry refers to the systematic process of extracting relevant data from individual studies included in the meta-analysis. It involves accurately recording key variables, such as study characteristics, sample sizes, effect sizes, and other relevant statistics, into a standardised format for subsequent analysis and synthesis.
Step 5: Data cleaning;Data cleaning involves amending incorrect, missing, corrupted, incorrectly formatted, duplicated, or incomplete data within a dataset.
In addition to data cleaning, we should also check our distribution of effect sizes. This involves:
checking the overall shape of the distribution of effect sizes
noticing anything abnormal about shape:
normal distribution?
predominantly small or large samples, or variety?
Also, be aware that each effect size from each study will have been derived from differing sample sizes and will have differing degrees of accuracy as estimates of the population/’true’ effect size.
To counter this, we ‘weigh’ each effect size:
by N (okay)
by inverse variance (which is preferable as it takes into account that effect sizes with smaller SEs are more precise estimates of the effect size, and while smaller SEs generally go hand in hand with larger samples, this is not always the case).
Step 6: Data analysis; systematically evaluating and combining data from multiple studies to draw statistical conclusions and identify patterns or trends.
Advantages and disadvantages
ADVANTAGES;
1.Handle a larger number of studies.
2.Improve preceision-smaller studies in isolation often insufficient to judge evidence of intervention effects.
3.To answer questions not posed by individual studies.
4. to settle controversies of results.Allows reasons for conflict to be assessed and explored.
DISADVANTAGES;
1.Very labour intensive.
2.Comparable vs incomparable. Sometimes values or constructs can be viweded similarly, and sometimes not.
3. Methodological quality;arguable which papes might meet certain high standards. Should they be included??
4. Number of studies.Need many many so unable to really delve deeply into the finer points of some studies.
5.There is a difference b/n the studies you were able to get, those you had a vague notion of but could not get, and those you were never even aware of.
representing meta analytic data.
While there are several ways to represent data visually, we will look at two representations: forest plots and weighted scatterplots.
Fixed effects vs random effects
FIXED EFFECTS MODELS;
-assumes all study effect sizes are drawn fro a population with a single “true” effect size
-considers variability b/n study results to be exclusively due to random variation or within-study variation.
-between study variation which is systematic is due to other factors which we can potentially examine via a further anlysis.
-possibly better if small number of studies
RANDOM EFFECTS MODELS
-more conservative
-assumes study effect sizes are drawn from multiple populations with potentially differing “true” effect sizes
-takes this into account by an additional weighting factor (as well as inverse variance of each effect size) which reflectsthe heterogeneity of the set of effect sizes
-b/n studies variation is random and from unidentified sources.
WHICH MODEL DO I USE?:
-no consensus on which is better
-a substantial difference in the combined effect calculated by the fixed and random models will be seen if studies are markedly heterogenous
-the confidence interval for a random effects model effect size will be wider and hence the model is more conservative when determining sig of the combined effect size. (so which you prefer depends on if your focus is the magnitude of the effect or the significance of the effect)
-if heterogeneity exists you can either choose to use a random effects model or look for the source of the heterogeneity via further analysis.
-may run both styles and compare results
determining effect size
SO HOW DO I DETERMINE IF MY EFFECT SIZES ARE HETEROGENOUS?
A. Cochrane’s Q Homogeneity Test;
-tests whether effect sizes included are related to/estimate the one population effect size
-sig results suggests the set of effect sizes (and hence the population itself) being tested is heterogenous and use of random-effects model or sub-group analyses is warranted
-more common in met- analyses
B. I squared Index:
-percentage of study variation due to heterogeneity
-calculated from the Q statistic and its degrees of freedom
- 100% x (Q-df)/Q
- rules of thumb vary but if it is over 50% you probably should be investigating further.
checking for heterogeneity
SUB GROUP/SET META ANALYSIS;
-check for moderators to explain heterogeneity of study effect sizes
-group studies/effect sizes into sub groupsand run Meta analysison them
-run heterogeneity tests on these anlyses to see if it is still significant.
further
FURTHER ANALYSES WITH EFFECT SIZES
-once you have your set of effect sizes, you can use a range of different analyses to test for moderation, group differences etc (regression, anova/ancova etc)
-using effect sizes as dependent variable (ie each is a data point, then can analyse these as you wish).
-limited only by available data and your imagination
meta regressions & representations of meta-analytic findings
META-REGRESSIONS;
-either a correlation or regression using the effect sizes as the dv and some kind of study attribute as the predictor variable.
-effect sizes weighted by sample size or inverse variance
-can do simple bivariate and also multivariate depending on available data
eg % sample meeting some criteria (eg gender, dx etc)
eg timing element (year, delay b/n measurement periods etc)
eg age of study participants
eg number of treatments or intervention sessions
graphical representations of meta-analytic findings
FOREST PLOTS;
-effect sizes & error bars (CI’s)
-allows visual representation and sig findings (individually and b/n each study)
-weghting of each study (and accuracy of associated effect size estimate) denoted by confidence interval width(wider the confidence interval, the less weighting a study has)
WEIGHTED SCATTERPLOTS;
-allows reader to see visually where the weighting of the studies comes into play in the analysis
eg of a forest plot
the forest plot was looking at many studies who had considered depression in holocaust survivors to controls.
vertical line shows effect size of zero. studies to the left of the zero line showed holocust survivors having lower depression scores than control, whereas on the right was the more typical finding of hocoaust survivors scoring higher than controls.
“overall” shows the combined effect size over all the studies.
N= total number of participants.
Fails safe N=number of studies with the opposite finding to the Overall Effect, which would be required to overturn the finding of the Overall Effect.
In the eg plot, the studies have been sorted by effect size, but you could sort via many other methods, such as sample size, year of study
forest plot eg 2
Top 2 rows show 1 hs=1 parent was a holocaust survivor, vs 2hd parents= both parents holocaust survivors.
Also shows studies with all female participants found larger effect size compared with all male participants etc.
Have also been able to compare eg MMPI measures against Other Measures etc which is also very useful. ie depending on measures used, the effect size altered quite a lot.
weighted scatterplot eg
ie “pom poms” have many more participants than justa fe spikes, and so have greater weighting.
hidden or unobtainable research
When you find yourself missing data, try the following:
-contacting original authors, who may still have relevant data or information on file
-calculating the data yourself, e.g. no descriptives but p or t given, you can convert to a d/g
MULTIPLE RELEVANT RESULTS;
Something to bear in mind for a meta-analysis is that each effect size you include needs to have come from a different group of people (which is related to the assumption of independence), as no single effect size should be coming from the same group.
There are two scenarios where you get multiple results in the same paper. One is known as conceptual replication, which is where participants in the study complete measures of the same variable. This is often used in kinds of psychometric assessments. In this instance, you can create an average effect size across the effect sizes related to that variable so that you’re only the contributing one.
The other scenario where you might get multiple results is what’s known as a fully replicated design. A common example of this is where statistics may be reported just for subgroups rather than for a whole group, i.e. if one large group is split into two age brackets (18–45 and 46–75+), they may have results for each age bracket but not for the total testing group.
UNOBTAINABLE RESEARCH
“file drawer research”. Bits that were never published etc.
To combat this, do a Fail Safe N Analysis=calculates how many opposing results to your findings, would be required to turn your sig finding to not sig.
PUBLICATION BIAS
remeber there is likely to be a publication bias where more likely to publish research findings with sig results.
Also check are not looking at the same study in just a re-published /alternate source.
