VOLUME 19 NUMBER 2 (July to December 2026)

Search for truth through data: NP decision processes, ROC functions, P-Functionals, knowledge updating, and sequential learning

Strong and vehement criticisms of statistical decision-making procedures, especially those using P-values and a level of significance (LoS) of α = .05, have abounded in recent years and is still continuing. For the sake of the scientific endeavor and the search for truth through data, there is tremendous impetus to re-examine and to improve these statistical decision-making procedures. This paper re-visits the fundamental problem of deciding the truth, based on data, between two competing hypotheses: a null H₀ and an alternative H₁. The Neyman-Pearson (NP) most powerful (MP) decision function, together with its power, remains the linchpin of statistical decision-making. Associated with this procedure is a decision process and a receiver operating characteristic (ROC) function. It is proposed that in reporting the outcome of the NP MP decision function, for a specified LoS, that it should always be accompanied by the value of the (equivalently, logarithm) likelihood ratio based on the decision function. The P-functional, which is the usual P-value statistic, associated with the decision process is re-examined. It is pointed out that P could be used in an equivalent implementation of the NP MP decision function, but if one wants to use its value to quantify the magnitude of support for either H₀ or H₁, then it should be the value of its (equivalently, logarithm) density function under H₁, which is the derivative of the ROC function, which should be reported. This will avoid the fallacy that smaller values of P are more supportive of H₁. Replicability of results is discussed in the context of realizations of the decision function or the P-functional. It is demonstrated that a coherent manner of acquiring knowledge about H₀ and H₁ is via sequential learning through Bayesian updating. But, it is also shown that publication bias could lead sequential updating astray in determining which of H₀ or H₁ is true. It is argued that a decision-maker can choose his own LoS, instead of using the conventional LoS of α = .05, since the summary measures accompanying the realized decision take into account the chosen LoS. Since a decision-maker is then free to choose his LoS, the question of how to choose it optimally arises. Three approaches for choosing the LoS are discussed, each appropriate for a specific situation a decision-maker encounters. A new approach to sample size determination is also described. The ideas are illustrated using concrete problems and a re-examination of Fisher’s lady tea-tasting experiment. It is hoped that recommendations under this fundamental setting will extend to complex settings and lessen criticisms of methods relying on P-values and an LoS of α= .05.