Teaching receiver operating characteristic analysis: an interactive laboratory exercise

Understanding decision curve analysis in clinical prediction model research

BACKGROUND

Many medical graduate students lack a thorough understanding of decision curve analysis (DCA), a valuable tool in clinical research for evaluating diagnostic models.

METHODS

This article elucidates the concept and process of DCA through the lens of clinical research practices, exemplified by its application in diagnosing liver cancer using serum alpha-fetoprotein levels and radiomics indices. It covers the calculation of probability thresholds, computation of net benefits for each threshold, construction of decision curves, and comparison of decision curves from different models to identify the one offering the highest net benefit.

RESULTS

The paper provides a detailed explanation of DCA, including the creation and comparison of decision curves, and discusses the relationship and differences between decision curves and receiver operating characteristic curves. It highlights the superiority of decision curves in supporting clinical decision-making processes.

CONCLUSION

By clarifying the concept of DCA and highlighting its benefits in clinical decisionmaking, this article has improved researchers' comprehension of how DCA is applied and interpreted, thereby enhancing the quality of research in the medical field.

Correlation of rheoencephalogram and intracranial pressure: results of a rat study

Measuring brain electrical impedance (rheoencephalography-REG) is a potential technique for noninvasive, continuous neuro-monitoring. Typically, intracranial pressure (ICP), an invasive monitoring modality, is used in brain monitoring. Our hypothesis was that both modalities would reflect cerebrovascular reactivity. In the present study we compared results of REG to results of ICP measurement. Rats were used under anesthesia ([Formula: see text]; 36 control and 59 vinpocetine infusions). REG was measured by two bipolar REG amplifiers; time constants (Tc) were 3 and 0.3 s. The vinpocetine injection caused a transient decrease in systemic arterial pressure (SAP) and a simultaneous increase in ICP and REG pulse amplitude. SAP decrease was 25%  ±  14%; ICP was 28%  ±  16%; REG pulse amplitude increase was 209%  ±  17% (Tc 3) and 107%  ±  68% (Tc 0.3). ICP increase correlated with REG pulse amplitude increase. Area under the receiver operating characteristic curve was 0.9481 for ICP-REG time constants 3 and 0.9335 for ICP-REG time constants 0.3; both with [Formula: see text]. The fact that both REG and ICP reflect cerebrovascular reactivity indicates the usefulness of REG as a potential technique for noninvasive, continuous neuro-monitoring. The Tc of REG amplifier requires optimization for continuous monitoring of pressure reactivity index.

Exploration of analysis methods for diagnostic imaging tests: problems with ROC AUC and confidence scores in CT colonography

BACKGROUND

Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.

METHODS

In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.

RESULTS

Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.

CONCLUSIONS

The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.

Offline eLearning for undergraduates in health professions: A systematic review of the impact on knowledge, skills, attitudes and satisfaction

BACKGROUND

The world is short of 7.2 million health-care workers and this figure is growing. The shortage of teachers is even greater, which limits traditional education modes. eLearning may help overcome this training need. Offline eLearning is useful in remote and resource-limited settings with poor internet access. To inform investments in offline eLearning, we need to establish its effectiveness in terms of gaining knowledge and skills, students' satisfaction and attitudes towards eLearning.

METHODS

We conducted a systematic review of offline eLearning for students enrolled in undergraduate, health-related university degrees. We included randomised controlled trials that compared offline eLearning to traditional learning or an alternative eLearning method. We searched the major bibliographic databases in August 2013 to identify articles that focused primarily on students' knowledge, skills, satisfaction and attitudes toward eLearning, and health economic information and adverse effects as secondary outcomes. We also searched reference lists of relevant studies. Two reviewers independently extracted data from the included studies. We synthesized the findings using a thematic summary approach.

FINDINGS

Forty-nine studies, including 4955 students enrolled in undergraduate medical, dentistry, nursing, psychology, or physical therapy studies, met the inclusion criteria. Eleven of the 33 studies testing knowledge gains found significantly higher gains in the eLearning intervention groups compared to traditional learning, whereas 21 did not detect significant differences or found mixed results. One study did not test for differences. Eight studies detected significantly higher skill gains in the eLearning intervention groups, whilst the other 5 testing skill gains did not detect differences between groups. No study found offline eLearning as inferior. Generally no differences in attitudes or preference of eLearning over traditional learning were observed. No clear trends were found in the comparison of different modes of eLearning. Most of the studies were small and subject to several biases.

CONCLUSIONS

Our results suggest that offline eLearning is equivalent and possibly superior to traditional learning regarding knowledge, skills, attitudes and satisfaction. Although a robust conclusion cannot be drawn due to variable quality of the evidence, these results justify further investment into offline eLearning to address the global health care workforce shortage.

Optimal thresholds by maximizing or minimizing various metrics via ROC-type analysis

RATIONALE AND OBJECTIVES

Based on imaging features, the optimal thresholds are typically determined as cutoff points to dichotomize the corresponding measurement scales.

MATERIALS AND METHODS

Five metrics (ie, the Youden index, Euclidian distance, percent of correct diagnosis, kappa statistic, and mutual information) are individually maximized or minimized to derive the corresponding optimal threshold. These optimal thresholds are estimated under the parametric binormal assumption. Monte Carlo simulation studies are conducted to compare the performances of these different methods. A published radiological example on the choice of treatment outcomes following ureteral stones is used to illustrate and compare the estimated thresholds both empirically and parametrically.

RESULTS

The optimal threshold can be a "moving target" because it would depend on modeling assumptions, metrics, and variability in the data. Even with large samples, disease prevalence has an impact on the robustness of the metrics.

CONCLUSIONS

It is recommended that researchers compare different optimal cutoff points using several metrics and select one that is most clinically relevant. The ultimate goal is to maximize diagnostic performances that are clinically meaningful to achieve improved global health.

Combined diffusion-weighted, blood oxygen level-dependent, and dynamic contrast-enhanced MRI for characterization and differentiation of renal cell carcinoma

PURPOSE

To investigate a multiparametric magnetic resonance imaging (MRI) approach comprising diffusion-weighted imaging (DWI), blood oxygen-dependent (BOLD), and dynamic contrast-enhanced (DCE) MRI for characterization and differentiation of primary renal cell carcinoma (RCC).

MATERIAL AND METHODS

Fourteen patients with clear-cell carcinoma and four patients with papillary RCC were examined with DWI, BOLD MRI, and DCE MRI at 1.5T. The apparent diffusion coefficient (ADC) was calculated with a monoexponential decay. The spin-dephasing rate R2* was derived from parametric R2* maps. DCE-MRI was analyzed using a two-compartment exchange model allowing separation of perfusion (plasma flow [FP] and plasma volume [VP]), permeability (permeability surface area product [PS]), and extravascular extracellular volume (VE). Statistical analysis was performed with Wilcoxon signed-rank test, Pearson's correlation coefficient, and receiver operating characteristic curve analysis.

RESULTS

Clear-cell RCC showed higher ADC and lower R2* compared to papillary subtypes, but differences were not significant. FP of clear-cell subtypes was significantly higher than in papillary RCC. Perfusion parameters showed moderate but significant inverse correlation with R2*. VE showed moderate inverse correlation with ADC. Fp and Vp showed best sensitivity for histological differentiation.

CONCLUSION

Multiparametric MRI comprising DWI, BOLD, and DCE MRI is feasible for assessment of primary RCC. BOLD moderately correlates to DCE MRI-derived perfusion. ADC shows moderate correlation to the extracellular volume, but does not correlate to tumor oxygenation or perfusion. In this preliminary study DCE-MRI appeared superior to BOLD and DWI for histological differentiation.