What a Well-Designed Negative Result Shows

ARRIVE relates sample-size justification to the primary variable, relevant effect and anticipated variability. Without that reference, readers cannot judge whether an experiment was sized for a demanding question or only to detect very large differences. ARRIVE document.

In a hypothetical example, a study designed to detect a large difference cannot necessarily rule out a small one. Giving both the same negative-result label does not give them the same scope.

Dienes distinguishes data more compatible with a no-effect explanation from data insufficiently sensitive to separate explanations. A comparison may fail to reach significance while remaining compatible with effects of interest. The article examines ways to evaluate that difference, including Bayesian evidence. Methodological paper.

The conclusion should reflect the available sensitivity. If estimates are imprecise, stating that the experiment did not distinguish these possibilities provides more information than declaring a general absence.

Equivalence tests assess whether data are incompatible with effects exceeding previously justified bounds. Lakens presents this approach using bounds based on the smallest difference of interest. Lack of significance in a conventional test does not replace that evaluation. Methodological paper.

Bounds should not be selected afterward to accommodate the result. Their interpretation depends on a scientific reason for considering differences between them small, as well as on the assumptions of the analysis used.

A Bayes factor compares predictions from specified models. It can help distinguish relative support for a no-effect explanation from data that barely discriminate, but its meaning depends on the alternative considered. It is not an automatic replacement for a p value or a measure independent of assumptions.

As a hypothetical example, a theory predicting enormous differences may be challenged by a precise result near zero, while another predicting small differences remains open. They should not be summarized as a single alternative hypothesis without being described.

An informative report presents the estimated difference, its uncertainty, the original question and the criterion used to interpret relevant magnitudes. It also explains how sample size was decided and which design features limit the conclusion.

A result can rule out a specific expectation under the studied conditions without demonstrating absence in every species, preparation or time period. That limit does not diminish its value: it identifies which part of an explanation is no longer compatible and which still needs another comparison.