Confidence Calibration: An Introduction With Application to Quality Improvement

Plasma metabolic fingerprints for large-scale screening and personalized risk stratification of metabolic syndrome

Direct diagnosis and accurate assessment of metabolic syndrome (MetS) allow for prompt clinical interventions. However, traditional diagnostic strategies overlook the complex heterogeneity of MetS. Here, we perform metabolomic analysis in 13,554 participants from the natural cohort and identify 26 hub plasma metabolic fingerprints (PMFs) associated with MetS and its early identification (pre-MetS). By leveraging machine-learning algorithms, we develop robust diagnostic models for pre-MetS and MetS with convincing performance through independent validation. We utilize these PMFs to assess the relative contributions of the four major MetS risk factors in the general population, ranked as follows: hyperglycemia, hypertension, dyslipidemia, and obesity. Furthermore, we devise a personalized three-dimensional plasma metabolic risk (PMR) stratification, revealing three distinct risk patterns. In summary, our study offers effective screening tools for identifying pre-MetS and MetS patients in the general community, while defining the heterogeneous risk stratification of metabolic phenotypes in real-world settings.

Development and validation of questionnaire-based machine learning models for predicting all-cause mortality in a representative population of China

Background

Considering that the previously developed mortality prediction models have limited applications to the Chinese population, a questionnaire-based prediction model is of great importance for its accuracy and convenience in clinical practice.

Methods

Two national cohort, namely, the China Health and Nutrition Survey (8,355 individual older than 18) and the China Health and Retirement Longitudinal Study (12,711 individuals older than 45) were used for model development and validation. One hundred and fifty-nine variables were compiled to generate predictions. The Cox regression model and six machine learning (ML) models were used to predict all-cause mortality. Finally, a simple questionnaire-based ML prediction model was developed using the best algorithm and validated.

Results

In the internal validation set, all the ML models performed better than the traditional Cox model in predicting 6-year mortality and the random survival forest (RSF) model performed best. The questionnaire-based ML model, which only included 20 variables, achieved a C-index of 0.86 (95%CI: 0.80-0.92). On external validation, the simple questionnaire-based model achieved a C-index of 0.82 (95%CI: 0.77-0.87), 0.77 (95%CI: 0.75-0.79), and 0.79 (95%CI: 0.77-0.81), respectively, in predicting 2-, 9-, and 11-year mortality.

Conclusions

In this prospective population-based study, a model based on the RSF analysis performed best among all models. Furthermore, there was no significant difference between the prediction performance of the questionnaire-based ML model, which only included 20 variables, and that of the model with all variables (including laboratory variables). The simple questionnaire-based ML prediction model, which needs to be further explored, is of great importance for its accuracy and suitability to the Chinese general population.

Thinking without knowing: Psychological and behavioral consequences of unjustified confidence regarding blackjack strategy

In two studies, we explored potential psychological and behavioral consequences of unjustified confidence, including outcome expectations, anxiety, risk taking, and information search and consideration. Study 1 employed an individual-differences approach to examine how participants' confidence regarding their knowledge of blackjack strategy, controlling for their actual knowledge, correlated with these hypothesized psychological and behavioral variables. Study 2 manipulated participants' confidence levels to examine these effects. Across the two studies, greater unjustified confidence led to larger bets (a measure of risk taking) and reduced use of hints designed to improve play (information search and consideration). Unjustified confidence also increased participants' outcome expectations and lowered anxiety levels. Implications of these findings, such as for educational interventions, are discussed.

Improving the Relationship Between Confidence and Competence: Implications for Diagnostic Radiology Training From the Psychology and Medical Literature

The focus of diagnostic radiology training is on creating competent professionals, whereas confidence and its calibration receive less attention. Appropriate confidence is critical for patient care both during and after training. Overconfidence can adversely affect patient care and underconfidence can create excessive costs. We reviewed the psychology and medical literature pertaining to confidence and competence to collect insights and best practices from the psychology and medical literature on confidence and apply them to radiology training. People are rarely accurate in assessments of their own competence. Among physicians, the correlation between perceived abilities and external assessments of those abilities is weak. Overconfidence is more prevalent than underconfidence, particularly at lower levels of competence. On the individual level, confidence can be calibrated to a more appropriate level through efforts to increase competence, including sub-specialization, and by gaining a better understanding of metacognitive processes. With feedback, high-fidelity simulation has the potential to improve both competence and metacognition. On the system level, systems that facilitate access to follow-up imaging, pathology, and clinical outcomes can help close the gap between perceived and actual performance. Appropriate matching of trainee confidence and competence should be a goal of radiology residency and fellowship training to help mitigate the adverse effects of both overconfidence and underconfidence during training and independent practice.

Measuring and Teaching Confidence Calibration Among Radiologists: A Multi-Institution Study

OBJECTIVE

Our purpose was to assess the calibration of resident, fellow, and attending radiologists on a simple image classification task (presence or absence of an anterior cruciate ligament [ACL] tear based on interpretation of sagittal proton density, fat-saturated MR images) and to assess whether teaching residents could improve their calibration.

METHODS

We created a test containing 30 randomized, sagittal, proton density, fat-saturated MR images of the ACL (15 normal, 15 torn). This test was administered in person to 20 trainees and 3 attendings at one medical center in one state. An online version of the test was given to 23 trainees and 14 attendings from 11 other medical centers in nine other states. Subjects were asked to give their confidence level (0%-100%) that each ACL was torn.

RESULTS

Cross-sectional data were collected from 60 radiologists (mean time after medical school = 9.3 years, minimum = 1 year, maximum = 36 years). This demonstrated a statistically significant improvement in calibration as a function of increasing experience (P = .020). Longitudinal data were collected from 12 trainees at the start and end of their musculoskeletal radiology rotation, with an intervening review of the primary and secondary signs of ACL tear on MR. A statistically significant improvement in calibration was noted during the rotation (P = .028).

CONCLUSIONS

Confidence calibration is a promising tool for quality improvement and radiologist self-assessment. Our study showed that calibration loss improves with experience in radiologists tested on a common and clinically important image classification task. We also demonstrated that calibration can be successfully taught to residents over a relatively short period (2-4 weeks).