An introduction to Meta-analysis

Chapter 1 How a meta-analysis works

Individual studies

Effect size

The effect size is a value which reflects the magnitude of the treatment effect or the strength of a relationship between two variables. It can represent any relationship between two variables. It is the unit of currency in a meta-analysis.

You compute the effect size for each study, and then work with the effect sizes to assess the consistency of the effect across studies and to compute a summary effect.

In graphs, the effect size for each study is represented by a square, with the location of the square representing both the direction and magnitude of the effect.

Precision

In a schematic, the effect size for each study is bounded by a confidence interval. This reflects the precision with which the effect size has been estimated in that study.

Study weights

In a schematic, the size of each square reflects the weight that is assigned to the corresponding study. There is a relationships between a study’s precision and that study’s weight in the analysis. Studies with good precision are assigned more weight. Precision is primarily driven by simple size.

Other elements can be used as well to assign weights.

p-values

For each study, a p-value for a test of the null is shown. A p-value will fall under .005 only if the 95% confidence interval does not include the null value.

The summary effect

Typically, you report the effect size of a summary effect, as well as a measure of precision and a p-value.

Effect size

In a plot, a summary effect is shown on the bottom line. It is the weighted mean of the individual effects. The mechanism used to assign the weights depends on our assumptions about the distribution of effect sizes form which the studies were sampled. 1) Fixed-effect model, there is assumed that all studies in the analysis share the same true effect size. The summary effect is the estimate of this common effect size 2) Random-effects model, there is assumed that the true effect sizes vary from study to study. The summary effect is our estimate of the mean of the distribution of effect sizes

Precision

The summary effect is represented by a diamond. The location represents the effect size while its width reflects the precision of the estimate. The precision addresses the accuracy of the summary effects as an estimate of the true effect.

P-value

The p-value reflects both the magnitude of the summary effect size and the volume of information of which the estimate is based.

Heterogeneity of effect sizes

The treatement effect is not always consistent across studies. If the effect size is consistent, we usually focus on the summary effect. If the effect sizevaries, we might still report the summary effect but note that the true effect in any given study could be somewhat lower or higher than this value. If it varies to heavily, the attention will go to the dispersion itself.

The dispersion in observed effects is partly spurios (it includes both real difference in effects and random error). Before trying to interpret this, we need to dermine what part of the observed variation is real.

Chapter 2 Why perform a meta-analysis

Introduction

The goal of a synthesis is to understand the results of any study in the context of all the other studies. We need no know whether the effect size is consistent across the body of data. If it is, we want to estimate the effect size as accureately as possible. If it isn’t we want to quantify the extent of variance and consider the implications.

Meta-analysis work with effect-sizes to determine whether or not the effect is consistent across studies. All of the are included in a single statistical synthesis.

Chapter 3 Overview

Treatment effects and effect sizes

The term effect size is appropriate when the index is used to quantify the relationship between two variables or a difference between two groups.

The term treatment effect is appropriate only for an index used to quantify the impact of a deliberate intervention.

Single group summary is when a meta-analysis has the goal of estimating a mean or risk or rate in a single population.

How to choose an effect size

Three considerations drive the choice of an effect size index are: 1) The effect sizes from different studies should be comparable to one another. They should measure (at least approximately) the same thing. 2) Estimates of the effect size should be computable from the information that is likely to be reported in published research reports. It should not require a re-analysis of the raw data 3) The effect size should have good technical properties

The effect size should be substantively interpretable.

The kind of data used in the primary studies will usually lead to a pool of two or three effect sizes that meet these criteria. 1) Based on means and standard deviations in two groups, the effect size will be raw difference in means, the standardized difference in means, or the response ratio. 2) Based on a binary outcome, the effect size will be the risk ratio, odds ratio, or risk difference. 3) Based on a correlation, the correlation can serve as an effect size

Parameters and estimates

An underlying effect size parameter is q. The sample estimate of that parameter is Y.

If a study has an infinitely large sample size, the it would yield an effect size Y that is identical to q. Because this isn’t possible, the sample estimate always differs. The value of Y  will vary from sample to sample. The distribution of these values is the sampling distribution of Y.

Chapter 4 Effectsizes based on means

Computing d and g from studies that use independent groups

We can estimate the standardized mean difference (d) from studies that used two independent groups as:

d = (Ẍ₁ - Ẍ₂) _/ s_within

Ẍ₁ and Ẍ₂ are the sample means in the two groups. The s_within  is the within-groups standard deviation, pooled across groups.

S_within = √ ((n₁ – 1) s²₁  + (n₂ – 1) s²₂ )/n₁+n₂ - s). 

N₁ and n₂ are the sample sizes in the two groups.
s₁ and s₂ are the standard deviations in the two groups.

The sample estimate of the standardized mean difference is Cohen’s d.

The variance of d is given by:

V_d = (n₁ + n₂/ n₁n₂) + d² /(2(n₁ + n₂))

The standard error of d is the square root of V_d

SE_d = √V_d

d tends to overestimate the absolute value of d in small samples. This can be removed by a correction that yields an unbiased estimate of d, this is sometimes called Hedges’g. To convert from d to g we use a correction factor J.

J = 1 - 3/4df -1.

Df is the degrees of freedom used to estimate s_within.
For two dependent groups, this is n₁ + n₂ - 2.

G = J * d

V_g = J² x V_d

SE_g = √V_d

J is always less than 1.0.
g will always be less than d in absolute value and variance.
J will be very close to 1.0 unless df is very small.

Chapter 8 Factors that affect precision

Variance, standard error and confidence intervals

The variance is a measure of the mean squared deviation from the mean effect. The variance for an effect size Y is V_y.

The computation of the variance is different for every effect size index. It metric is based on squared value.

The standard error is on the same scale as the effect size.
SE_{y =} √V_y.

If we assume that the effect size is normally distributed, we can compute a 95% confidence interval using

LL_y = Ῡ - 1.96 x SE_y

UL_y = Ῡ + 1.96 x SE_y

Z_y =  Ῡ /SE_y

Factors that affect precision

Factors that affect precision are: 1) sample size, larger studies yield more precise estimates than smaller samples. 2) Study design

Chapter 10 overview

Nomenclature

A study’s true effect size is the effect size in the underlying population. A study’s observed effect size is the effect size that is actually observed.

In the schematics this is represented as 1) Individual studies, circle for the true effect and a square for the observed effect 2) summary effects triangle for the true effect and a diamond for the observed effect

Fixed-effect model

The true effect size

Under the fixed-effect model, we assume that all studies in the meta-analysis share a common (true) effect size.

Impact of sampling error

The observed effect size varies from one study to the next only because of the random error inherent in each study.

The observed effect Y_i for any study is given by the population mean plus the sampling error in that study.

Y_i = q + error_i

We can estimate the sampling distribution of the errors.

Performing a fixed-effect meta-analysis

In a meta-analysis, we start with the observed effects and try to estimate the population effect. In order to obtain the most precise estimate of the population effect (to minimize the variance) we compute a weighted mean. The weight assigned to each study is the inverse of that study’s variance.

W_i = 1/ V_yi

V_yi is the within-study variance for study (i).

The weighted mean (M) is computed as:

M = ∑^k_i=1w_iy_i / ∑^k_i=1w_i

The sum of the products W_iY_i divided by the sum of the weights.

The variance of the summary effect is estimated as the reciprocal of the sum of the weights

V_m = 1 /∑^k_i=1w_i

SE_M = √V_M

LL_y = M - 1.96 x SE_M

UL_y = M + 1.96 x SE_M

Z_y =  M /SE_M

For a one-tailed test the p value is given by:

p = 1 - φ(±|Z|)

We use + if the difference is in the expected direction and - otherwise.
For a two-tailed test we use:

p = 2[1 - φ(|Z|) ]

 φ(Z) is the standard normal cumulative distribution.

Chapter 12 Random-effects model

The true effect sizes

In a random-effects meta-analysis, we assume that the true effects are normally distributed.

Impact of sampling error

If the true effect for a study is q_I, then the observed effect for that study will be less or greater than q_I, because of sampling error. The distance between the overall mean and the observed effect in any given study consists of two parts: true variation effect sizes z_i and the sampling error e_i

The observed effect Y_i for ay study is given by grand mean, the deviation of the study’s true effect from the grand mean, and the deviation of the study’s effect from the study’s true effect.

Y_i = m + z_I + e_i

The distance from m to each q_I depends on the standard deviation of the distribution of the true effects across studies, called tau (t). The same value of t² applies to all studies in the meta-analysis.

The distance from q_I to Y_i depends on the sampling distribution of the sample effects about q_I.  This depends on the variance of the observed effect size of each study.

Performing a random-effects meta-analysis

In a meta-analysis, we start with the observed effects and try to estimate the population effect. We use the collection of Y_i to estimate m. To obtain the most precise estimate of the overall mean we compute a weighted mean, where the weight assigned to each study is the inverse of that study’s variance.

We need to now the within-study variance and t².

Estimating tau-squared

T² is the between-studies variance. If we somehow new the true effect size of each study, and computed the variance of these effect sizes, this would be T². We estimate this with:

T² = (Q-df)/C

Q = ∑^k_i=W_iY²_i - ((= ∑^k_i=W_iY²)²/= ∑^k_i=W_i

Df = k -1

K is the number of studies

C = = ∑W_i -∑W²_i/ ∑W_i

Estimating the mean effect size

Each study will be weighted by the inverse of its variance. The variance includes the original (within-studies) variance plus the estimate of the between-studies variance T².

The weight assigned to each study is:
* is for the random effects model.

W_i *= 1/V*_Yi

V*_Yi = V_Yi + T²

M* = ∑^k_i=1w_iy_i / ∑^k_i=1w_i

V_m* = 1 /∑^k_i=1w_i*

SE_M* = √V_M*

LL_M* = M* - 1.96 x SE_M*

UL_M* = M* + 1.96 x SE_M*

Z_* =  M* /SE_M*

P* = 1 - φ(±|Z*|)

We use + if the difference is in the expected direction and - otherwise. For a two-tailed test we use:

P* = 2[1 - φ(|Z*|) ]

Chapter 13 Fixed-effect versus random-effects models

Definition of a summary effect

In the fixed-effect analysis, we assume that the true effect size is the same in all studies, and the summary effect tis our estimate of this common effect size.

In random-effects analysis, we assume that the true effect size varies from one study to the next, and that the studies in our analysis represent a random sample of effect sizes that could have been observed. The summary effect is an estimate of the mean of these effects.

Estimating the summary effect

Under the fixed-effect model, we assume that the only reason the effect size varies between studies is the sampling error. When assigning weights to different studies, we can largely ignore the information in the smaller studies.

Under random-effect model, we want to be sure that all effect sizes are represented in the summary estimate. The goal is to estimate the mean effect in a range of studies, we don’t want that overall estimate to be overly influenced by any one of them.

In graphs, the weight assigned to each study is reflected in the size of the box for that study. The fixed-effect model has a wide range of weights. The random-effects model has a narrow range.

Extreme effect size in a large study or a small study

Whenever T² is nonzero, the relative weights assigned under random effects will be more balanced than those assigned under fixed effects.

Confidence interval

Under the fixed-effect model, the only source of uncertainty is the within-study (sampling or estimation) error.

Under the random-effects model, the sources of uncertainty are the within-study error and the between-studies variance.

The variance, standard error, and confidence interval for the summary effect will be larger under the random-effects model.

Under the fixed-effect model, the standard error of the summary effect is given by:

SE_M = √(s²/k x n )

Under the random,effects model, the standard error of the summary effect is given by:

SE_M = √((s²/k x n) + (T²/k))

The null hypothesis

Under the fixed-effect model, the null hypothesis is that there is zero effect in every study.

Under the random-effects model, the null hypothesis is that the mean effect is zero.

Which model should we use?

The selection is based on the expectation about whether the studies share a common effect size.

Fixed effect

It makes sense to sue the fixed-effect model if two conditions are met: if you believe that all studies included in the analysis are functionally identical and the goal is to compute the common effect size for the identified population, and not to generalize

Random effects

The goal here is to generalize to a range of scenarios.

A cavecat

If the number of studies is very small, then the estimate of the between-studies variance T² will have poor precision. One option here is to report the separate effects and not report a summary effect. Another option is to perform fixed-effect analysis. This doesn’t allow to make inferences about the wider population. A third option is to take a Bayesian approach.

Model should not be based on the test for heterogeneity

The decision should be based on our understanding of whether or not all studies share a common effect size, and not the outcome of a statistical test.

Chapter 20 Meta-regression

Introduction

Meta-regression is the use of regression in a meta-analysis. The covariates are at the level of the study. The dependent variable is the effect size in the studies.

There is a need to assign a weight to each study and to select the appropriate model. The R² index must be modified for use in meta-analysis.

In order for the analysis to be meaningful, there is a need for an appropriately large ratio of studies to covariates.

Fixed-effect model

Assessing the impact of the slope

If we are working with a primary study and wanted to test the coefficient for significance we might use a t-test of the form

t = B/SE_B

B is the covariate.

In meta-analysis, the coefficient for any covariate and its standard error are based on groups of studies.

Z = B/SE_B