Statisticians are calling on their profession to abandon one of its most treasured markers of significance. But what could replace it?
Statistical arguments are a crucial part of decision-making in a modern society. The kind of decisions that governments and large companies must make all the time are governed by probabilities. In those circumstances of uncertain knowledge we need to reduce a cloud of unknowing to facts as hard and cold as hailstones that can be acted on, or even just used in arguments. But some of the most popular techniques for doing this are now under attack from within the profession.
The p value is supposed to measure whether the conclusions drawn from any given experiment or investigation of data are reliable. It actually measures how unlikely the observed result is compared with what would be expected as a result of random chance. Obviously this requires a sophisticated understanding of the results that chance might be expected to produce. This isn’t always available. To take one popular example, any calculation of how likely we are to be the only intelligent species in the universe depends absolutely on assumptions about the likelihood of intelligent species arising, which can’t be tested across a range of universes.