P-Value, Confidence Intervals, and Statistical Inference: A New Dataset of Misinterpretation

Comprehension of confidence intervals in audio-visual patient information materials for people with multiple sclerosis (COCO-MS): A web-based randomised controlled, parallel group trial

OBJECTIVES

To evaluate patient information materials on confidence intervals (CIs) in multiple sclerosis to be used with patient decision aids.

METHODS

Web-based randomised controlled parallel group trial with four study arms. Participants were equally allocated to one of three versions of audio-visual patient information or to a standard written information (arm IV). In the short version (arm III), CIs were explained without using an example, in the other two versions examples were used (arm I and arm II). The examples are based on an apple farmer who wants to estimate the average weight of his apples (arm I) and to test a treatment against worms (arm II). Primary endpoint was comprehension of CIs, assessed with a six-item multiple-choice questionnaire.

RESULTS

855 of 1068 (80 %) randomised participants completed the survey (71 % arm I, 73 % arm II, 87 % arm III, 90 % arm IV). The median of correctly answered questions on CIs was 4 out of 6 questions in arms I and II and 5 out of 6 questions in arm III. Compared to the standard information (arm IV), all the other arms scored better on the comprehension questionnaire (ANOVA, p ≤ 0.003).

CONCLUSIONS

Information about CIs can be presented comprehensibly. High scores and a high rate of completers indicate that the short version is the favourable one.

PRACTICE IMPLICATIONS

Information materials on CIs should be used alongside absolute risk reductions in patient decision aids to enhance the interpretation of study results.

Beyond psychology: prevalence of p value and confidence interval misinterpretation across different fields

P values and confidence intervals (CIs) are the most widely used statistical indices in scientific literature. Several surveys have revealed that these two indices are generally misunderstood. However, existing surveys on this subject fall under psychology and biomedical research, and data from other disciplines are rare. Moreover, the confidence of researchers when constructing judgments remains unclear. To fill this research gap, we surveyed 1,479 researchers and students from different fields in China. Results reveal that for significant (i.e., p < .05, CI does not include zero) and non-significant (i.e., p > .05, CI includes zero) conditions, most respondents, regardless of academic degrees, research fields and stages of career, could not interpret p values and CIs accurately. Moreover, the majority were confident about their (inaccurate) judgements (see osf.io/mcu9q/ for raw data, materials, and supplementary analyses). Therefore, as misinterpretations of p values and CIs prevail in the whole scientific community, there is a need for better statistical training in science.