Meta-analysis in mental health reserach
Cuipters, P (2016)

Advantages and problems of meta-analysis

Advantages of meta-analysis are: 1) The statistical power to detect effects is higher than for individual studies,this makes a more precise and accurate estimation of true effects possible 2) It is possible to explore inconsistencies between studies and to examine whether the effects of the intervention differs among specific subgroups of studies. 3) It is possible to make an estimate of the number of studies that were conducted but not published

Problems with meta-analyses are: 1) Garbage in, garbage out, they can never be better than the studies they summarize 2) They combine apples and oranges, there are always differences between studies. 3) The file drawer problem, not all relevant studies are published and are often not included in meta-analyses 4) Researcher allegiance, the agenda-driven bias of researchers who conduct the meta-analyses.

Publication bias

Publication bias is the problem that not all the studies that are conducted in a certain area are actually published. Publication of studies which show significant effect and large effects of intentions are favoured. This can lead to an over-estimation of the effect.

There are several other types of reporting bias 1)  Time lag bias, some studies are published later than others, depending on the nature and direction of the results 2) Outcome reporting bias 3) Language bias, when studies in another language are not identified and these studies differ in terms of nature and direction of the results.

Testing for publication bias with indirect methods: the funnel plot

In some cases it is possible to examine publication bias directly.

If it is not possible to compare published with unpublished trials, it is possible to get an indirect impression whether there is publication bias or not. These estimates are based on the assumption that large studies can make a more precise estimate of the effect size. Random variations of the effect sizes are larger in studies with few participants, a difference that can be represented graphically in a ‘funnel plot’. In this plot the effect size is represented at the horizontal axis and the size of the study on the vertical axis. If the effect sizes differ from the mean effect size only by chance, they should divert in both directions, both positive and negative.

There are several tests for the asymmetry of the funnel plot.

There is also a method to impute the missing studies and estimate the effect size after imputation of these missing studies.

Risk of the funnel plot: 1) It requires a considerable number of studies, at least 30, 2) It depends of the effect size of the studies 3) How the funnel plot looks depends on other factors like: the type of outcome, the parameter of the vertical axis, when heterogeneity is high 4) It allows to see if small studies with negative effects have not been published while they should have been published, but there may be other reasons why studies with small samples have larger effects than larger studies.

Sources of bias in randomized trials and assessing risk of bias

Selection bias

Selection bias are systematic differences between groups that were randomized in the trial. It can result from errors in the randomization process. Researchers should must not foresee the assignment.

Performance and detection bias

Performance bias are systematic differences between groups in the care that is provided. Can be prevented by blinding of participants and the personnel involved in the study. Detection bias refers to systematic differences between groups in how outcomes are determined. This can be prevented by blinding the outcome assessors.

Attrition bias

Analysing all randomized participants (also those who drop out) is important for estimating the true effect of an intervention. People who drop out of intervention can still participate in the assessment of outcome after the intervention.

Reporting bias

It is possible to verify outcomes in trial registries.

Other potential threats to validity

Another threat to validity is extreme baseline imbalance between the randomized groups.

Researcher allegiance

Researcher allegiance is a researcher’s belief in the superiority of a treatment (and) the superior validity of the theory of change that is associated with the treatment.

It is measured by checking several characteristics of the studies.

Assessing risk of bias: the Cochrane Risk of bias assessment tool

Reporting of risk of bias is important.