Bayesian Versus orthodox statistics: which side are you on? - summary of an article by Dienes, 2011

Critical thinking
Article: Dienes, Z, 2011
Bayesian Versus orthodox statistics: which side are you on?
doi: 10.1177/1745691611406920

The contrast: orthodox versus Bayesian statistics

The orthodox logic of statistics, starts from the assumption that probabilities are long-run relative frequencies.
A long-run relative frequency requires an indefinitely large series of events that constitutes the collective probability of some property (q) occurring is then the proportion of events in the collective with property q.

• The probability applies to the whole collective, not to any one person.
  • One person may belong to two different collectives that have different probabilities
• Long run relative frequencies do not apply to the truth of individual theories because theories are not collectives. They are just true or false.
  • Thus, when using this approach to probability, the null hypothesis of no population difference between two particular conditions cannot be assigned a probability.
• Given both a theory and a decision procedure, one can determine a long-run relative frequency with which certain data might be obtained. We can symbolize this as P(data| theory and decision procedure).

The logic of Neyman Pearson (orthodox) statistics is to adopt decision procedures with known long-term error rates and then control those errors at acceptable levels.

• Alpha: the error rate for false positives, the significance level
• Beta: the error rate for false negatives

Thus, setting significance and power controls long-run error rates.

• An error rate can be calculated from the tail area of test statistics.
• An error rate can be adjusted for factors that affect long-run error rates
• These error rates apply to decision procedures, not to individual experiments.
  • An individual experiment is a one-time event, so does not constitute a long-run set of events
  • A decision procedure can in principle be considered to apply over a indefinite long-run number of experiments.

The probabilities of data given theory and theory given data

The probability of a theory being true given data can be symbolized as P(theory|data).
This is what orthodox statistics tell us.
One cannot infer one conditional probability just by knowing its inverse. (So P(data|theory) is unknown).

Bayesian statistics starts from the premise that we can assign degrees of plausibility to theories, and what we want our data to do is to tell us how to adjust these plausibilities.

• When we start from this assumption, there is no longer a need for the notion of significance, p value, or power.
• Instead, we simply determine the factor by which we should change the probability of different theories given the data.

The likelihood

In the Bayesian approach, applies to the truth of theories.
We can answer the questions about:

• p(H), the probability of a hypothesis being true (our prior probability)
• p(H|D), the probability of a hypothesis given the data (our posterior probability).

Neither of these can be do using the orthodox approach.
Likelihood: the probability of obtaining the exact data given the hypothesis.

Posterior is given by likelihood times prior.

The likelihood principle: all information relevant to inference contained in data is provided by the likelihood.
When we are determining how given data changes the relative probability of our different theories, it is only the likelihood that connects the prior to the posterior.

The likelihood is the probability of obtaining the exact data obtained given a hypothesis (P(D|H).
This is different from a p value, which is the probability of obtaining the same or more extreme data given both a hypothesis and a decision procedure.

• A p-value for a t test is a tail area of the t distribution
• The corresponding likelihood is the height of the distribution at the point representing the data

In orthodox statistics, p values are changed according to the decision procedure; under what conditions one would stop collecting data, whether or not the test is post hoc, how many other test one conducted.
None of these factor influence the likelihood.

The Bayes factor

The Bayes factor pits one theory against another.

Prior probabilities and prior odds can be entirely personal and subjective.
There is no reason why people should agree about these before data are collected if they are not part of the publically presented inferential procedure.
If the priors form part of the inferential procedure, they must be fairly produced and subjected to the tribunal of peer judgement.

One data are collected we can calculate the likelihood for each theory.
These likelihoods are things we want researchers to agree on. Any probabilities that contribute to them should be plausibly or simply determined by determined by the specification of the theories.
The Bayes factor (B): the ratio of likelihoods.

Posterior odds = B x prior odds.

• If B is greater than 1, the data supported you experimental hypothesis over the null.
• If B is less than 1, the data supported the null hypothesis over the experimental one.
• If B is about 1, the experiment was not sensitive.

The evidence is continuous and there are not thresholds in Bayesian theory.
B automatically gives a notion of sensitivity, it directly distinguishes data supporting the null from data uninformative about whether the null or you theory was supported.

For both p values associated with a t test and for B, if the null is false, as a number of subjects increases, then test scores are driven in one direction.

• p values are expected to become smaller
• Both t and B values are expected to become larger

When the null hypothesis is true, p values are not driven in any direction, only B us. B is then driven to zero.

Problems with the Neyman Pearson approach

Stopping rule

In the Neyman Pearson approach, one must specify the stopping rule in advance.
Once those conditions are met, there is to be no more data collection.
Typically, this means one should use a power calculation to plan in advance how many subjects to run.

The Bayes factor behaves differently from p values as more data are run (regardless of stopping rule).

• For a p value, if the null is true, any value in the interval 0 to 1 is equally likely no matter how much data you collect
  • For this reason, sooner or later, you are guaranteed to get a significant result if you run subjects long enough and stop when you get the p value you want
  • When the null is true, as the number of subjects increases, the p value is not driven to any particular value.
• As the number of subjects increases and the null is true, the Bayes factor is driven toward zero.

Planned versus post hoc comparisons

When using Neyman Pearson, it matters whether you formulated your hypothesis before or after looking at the data (post hoc vs. planned comparisons).
Predictions made before rather than after looking at the data are treated differently.

• Post hoc fitting can involve preference for one auxiliary over may others of at least equal plausibility.

In Bayesian inference, the evidence for a theory is just as strong regardless of its timing relative to the data.
This is because the likelihood is unaffected by the time the data were collected.
The likelihood principle follows from the axioms of probability.
It is not the ability to predict in advance per se that is important, that ability is just an (imperfect) indicator of the prior probability of relevant hypotheses.
When performing Bayesian inference, there is no need to adjust for the timing of predictions per se.

Multiple testing

When using Neyman Pearson, one must correct for how many tests are conducted in a family of tests.

When using Bayes, it does not matter how many other statistical hypotheses are investigated. All that matters is the data relevant to each hypothesis under investigation.
Once one takes into account the full context, the axioms of probability lead to sensible answers.

In the Bayes approach, rather than the Neyman Pearson approach, that is most likely to demand that researchers draw appropriate conclusions from a body of relevant data involving multiple testing.

The rationality of the Bayesian approach

If we want to determine by how much we should revise continuous degrees of belief, we need to make sure our system of inference obeys the axioms of probability.
If researchers want to think in terms of degree of support data provide for a hypothesis, they should make sure their inferences obey the axioms of probability.

One version of degrees of belief are subjective probabilities.
Subjective probabilities: personal convictions in an opinion.
When probabilities of different propositions form part of the inferential procedure we use in deriving conclusions from data, then we need to make sure that the procedure is fair.
Thus, there has been an attempt to specify objective probabilities that follow from the informational specification of a problem.
In this way, the probabilities become an objective part of the problem, with values that can be argued about, given the explicit assumptions, and that do not depend any further on personal idiosyncrasies.

One notion of rationality is having sufficient justification for one’s beliefs.
If one can assign numerical continuous degrees of justification to beliefs, then some simple minimal desiderata lead to the likelihood principle of inference.
Hypothesis testing violates the likelihood principle.

Effect size

Bayes factors demand consideration of relevant effect sizes.

Neyman developed two specific measures of sensitivity:

• Power
• Confidence intervals: the set of population values that the data are consistent with.

For any continuous measure based on a finite number of subjects, an interval cannot be an infinitesimally small point.
A null result is always consistent with population values other than zero.
That is why a non-significant result cannot on its own lead to the conclusion that the null hypothesis is true.

Theories and practical questions generally specify, even if vaguely, relevant effect sizes.
The research context, usually provides a range of effects that are too small to be relevant and a range of effects that are consistent with theory or practical use.

Researchers have relevant intuitions, and that is why it has made sense to them to assert null hypotheses.
Bayes makes them explicit.
If we want to use null results in any way to count against theories that predict an effect, we must consider the range of effect sizes consistent with the theory.

Effect size is very important in the Neyman Pearson approach.

• One must specify the sort of effect one predicts in order to calculate power.

On the other hand, Fisherian significance testing leads people to ignore effect sizes.

• People have followed Fisher’s method, while paying lip service to effect sizes, but not heeding Fisher’s advice that nothing follows from a null result.

One must specify what sort of effect sizes a theory predicts to calculate a Bayes factor.
Because it takes into account effect size, the Bayes factor distinguishes evidence that there is not relevant effect from no evidence of a relevant effect.
One can only confirm a null hypothesis when one has specified the effect size expected on the theory being tested.

In specifying theoretically expected effect sizes, we should ask ourselves “What size effect does the literature suggest is interesting for this particular domain?” Rather than following the common practice of plucking a standardized effect size of 0.5 out of thin air, researchers should get to know the data of the field.

Confidence intervals themselves have all the problems for Neyman Pearson inference in general (unlike credibility or likelihood intervals).
Because confidence intervals consists of all values non-significantly different from the sample mean, they inherit the arbitrariness of significance testing.

How to calculate a Bayes factor

To calculate a Bayes factor in support of a theory, one has to specify what the probability of different effect sizes are, given the theory.
Bayes gives us the apparatus to flexibly deal with different degrees of uncertainty regarding the predicted effect size.
Logically, one needs to know what a theory predicts in order to know how much it is supported by evidence.

Three distributions

In terms of predictions of the theory (or requirements of a practical effect), one has to decide what range of effects are relevant to the theory.
Three ranges:

• An uniform distribution
  All values between a lower bound and an upper bound.
  All values are represented as possible and equally likely given the theory and all those outside are inconsistent with the theory
• A normal distribution
  One value is the most likely given the theory, and any values lower or higher are progressively less likely
• Normal distribution centred on zero with only one tail
  The theory predicts an effect in one direction, but smaller values are generally more likely than larger values

Different ways of using Bayes factors

Bayes factor is suggested to be used on any data where the null hypothesis is compared with a default theory.
Or when inference is based on the posterior and thus takes into account the priors of hypothesis.
Also for specific hypotheses that interest the researcher and allows priors to remain personal and not part of public inference.

By following Bayes rule, each of these approaches means rational answers are provided for the given assumptions, and researchers may choose each according to their goals and which assumptions seem relevant to them.

Bayes factors are just one form of Bayesian inference, namely a method for evaluating theories against another.

Multiple testing and cheating

With Bayes factors, one does not have to worry about corrections for multiple testing, stopping rules, or planned versus post hoc comparisons.
Bayes factor just tells you how much support given data provides for one theory over another.
There is no right Bayes factor.

Strictly, each Bayes factor is a completely accurate indication of the support for the data of one theory over another.
The theories are defined by the precise predictions they make.
The crucial question is which of these representations best matches the theory as the researcher has described it and related it to the existing literature.

One constraint on the researcher will be the demand for consistency: arguing for one application of a theory ties one’s hands when it comes to another application.
The solution is to use a default Bayes factor for all occasions, though this amounts to evaluating a default theory for all occasions, regardless of one’s actual theory.
A default Bayes factor will only test your theory if it happens to correspond to the default.
Another solution is to define the predictions according to simple procedures to ensure the theory proposed is tested according to fair criteria.

When using Bayes in multiple testing, one can use the fact that one is testing multiple hypotheses to inform the results if one believes that testing these multiple hypotheses is relevant to the probability of any of them being true.

Weaknesses of the Bayesian approach

• Bayesian analysis force people to consider what a theory actually predicts, but specifying the predictions in detail may be contentious.
• Bayesian analysis escape the paradoxes of violating the likelihood principle, but in doing so they no longer control for Type I and Type II errors.

Calculating a Bayes factor depends on answering the following question about which there may be disagreement: What way of assigning probability distributions of effect sizes as predicted by theories would be accepted by protagonists on all sides of a debate?

Ultimately, the issue is about what is more important to us: using a procedure with known long-term error rates or knowing the degree of support for our theory.