Prediction Intervals and Confidence Intervals

In NIST's regression example, the confidence interval describes uncertainty in the mean response at specified predictor values. It is not intended to contain all individual observations. Its frequentist interpretation concerns the procedure's coverage across repetitions under the model assumptions. NIST document.

A narrow band around a curve can therefore indicate a well-estimated mean without demonstrating that future measurements will be equally close to the curve.

For a new observation, NIST incorporates both uncertainty in the estimated mean and the residual variation of that observation. In the presented regression model, this produces a wider interval than the interval for the mean at the same point and coverage level. NIST document.

This is not a penalty for making predictions. It recognizes that an individual measurement can depart from the mean even when that mean is known quite precisely.

Consider a hypothetical example: one team wants to estimate the mean response of its procedure, while another needs to anticipate a new reading. Both may use the same fit, but they are asking different questions. The first examines a mean; the second must include variation among observations.

A future reading outside the confidence interval for the mean does not, by itself, demonstrate failure. Comparing it with that interval uses a reference constructed for a different quantity. The interval's complete label should accompany the graph.

Higgins and colleagues distinguished inference about the mean effect from inference about the distribution of effects across studies. When heterogeneity exists, a precise mean estimate does not guarantee that a new context will have a nearby effect. The article proposed ways to express that predictive uncertainty. Methodological paper.

Here, the target can be the underlying effect of a future study, not an individual laboratory observation. Calling both prediction intervals also requires stating which unit is being predicted.

Coverage depends on the model, how the data were obtained and what the new observation represents. Software can calculate apparently precise limits without checking whether the intended use matches the question for which they were constructed.

Before comparing intervals, identify the target, coverage level and predictor conditions. Also check whether the software returns uncertainty for the mean or for a new observation. Width becomes meaningful only after that difference is clear.