Assessing cognitive screeners with the critical success index

On the Dependence of the Critical Success Index (CSI) on Prevalence

The critical success index (CSI) is an established metric used in meteorology to verify the accuracy of weather forecasts. It is defined as the ratio of hits to the sum of hits, false alarms, and misses. Translationally, CSI has gained popularity as a unitary outcome measure in various clinical situations where large numbers of true negatives may influence the interpretation of other, more traditional, outcome measures, such as specificity (Spec) and negative predictive value (NPV), or when unified interpretation of positive predictive value (PPV) and sensitivity (Sens) is needed. The derivation of CSI from measures including PPV has prompted questions as to whether and how CSI values may vary with disease prevalence (P), just as PPV estimates are dependent on P, and hence whether CSI values are generalizable between studies with differing prevalences. As no detailed study of the relation of CSI to prevalence has been undertaken hitherto, the dataset of a previously published test accuracy study of a cognitive screening instrument was interrogated to address this question. Three different methods were used to examine the change in CSI across a range of prevalences, using both the Bayes formula and equations directly relating CSI to Sens, PPV, P, and the test threshold (Q). These approaches showed that, as expected, CSI does vary with prevalence, but the dependence differs according to the method of calculation that is adopted. Bayesian rescaling of both Sens and PPV generates a concave curve, suggesting that CSI will be maximal at a particular prevalence, which may vary according to the particular dataset.

Cardiovascular Diseases Diagnosis Using an ECG Multi-Band Non-Linear Machine Learning Framework Analysis

BACKGROUND

cardiovascular diseases (CVDs), which encompass heart and blood vessel issues, stand as the leading cause of global mortality for many people.

METHODS

the present study intends to perform discrimination between seven well-known CVDs (bundle branch block, cardiomyopathy, myocarditis, myocardial hypertrophy, myocardial infarction, valvular heart disease, and dysrhythmia) and one healthy control group, respectively, by feeding a set of machine learning (ML) models with 10 non-linear features extracted every 1 s from electrocardiography (ECG) lead signals of a well-known ECG database (PTB diagnostic ECG database) using multi-band analysis performed by discrete wavelet transform (DWT). The ML models were trained and tested using a leave-one-out cross-validation approach, assessing the individual and combined capabilities of features, per each lead or combined, to distinguish between pairs of study groups and for conducting a comprehensive all vs. all analysis.

RESULTS

the Accuracy discrimination results ranged between 73% and 100%, the Recall between 68% and 100%, and the AUC between 0.42 and 1.

CONCLUSIONS

the results suggest that our method is a good tool for distinguishing CVDs, offering significant advantages over other studies that used the same dataset, including a multi-class comparison group (all vs. all), a wider range of binary comparisons, and the use of classical non-linear analysis under ECG multi-band analysis performed by DWT.

Critical success index or F measure to validate the accuracy of administrative healthcare data identifying epilepsy in deceased adults in Scotland

BACKGROUND

Methods to undertake diagnostic accuracy studies of administrative epilepsy data are challenged by lack of a way to reliably rank case-ascertainment algorithms in order of their accuracy. This is because it is difficult to know how to prioritise positive predictive value (PPV) and sensitivity (Sens). Large numbers of true negative (TN) instances frequently found in epilepsy studies make it difficult to discriminate algorithm accuracy on the basis of negative predictive value (NPV) and specificity (Spec) as these become inflated (usually >90%). This study demonstrates the complementary value of using weather forecasting or machine learning metrics critical success index (CSI) or F measure, respectively, as unitary metrics combining PPV and sensitivity. We reanalyse data published in a diagnostic accuracy study of administrative epilepsy mortality data in Scotland.

METHOD

CSI was calculated as 1/[(1/PPV) + (1/Sens) - 1]. F measure was calculated as 2.PPV.Sens/(PPV + Sens). CSI and F values range from 0 to 1, interpreted as 0 = inaccurate prediction and 1 = perfect accuracy. The published algorithms were reanalysed using these and their accuracy re-ranked according to CSI in order to allow comparison to the original rankings.

RESULTS

CSI scores were conservative (range 0.02-0.826), always less than or equal to the lower of the corresponding PPV (range 39-100%) and sensitivity (range 2-93%). F values were less conservative (range 0.039-0.905), sometimes higher than either PPV or sensitivity, but were always higher than CSI. Low CSI and F values occurred when there was a large difference between PPV and sensitivity, e.g. CSI was 0.02 and F was 0.039 in an instance when PPV was 100% and sensitivity was 2%. Algorithms with both high PPV and sensitivity performed best in terms of CSI and F measure, e.g. CSI was 0.826 and F was 0.905 in an instance when PPV was 90% and sensitivity was 91%.

CONCLUSION

CSI or F measure can combine PPV and sensitivity values into a convenient single metric that is easier to interpret and rank in terms of diagnostic accuracy than trying to rank diagnostic accuracy according to the two measures themselves. CSI or F prioritise instances where both PPV and sensitivity are high over instances where there are large differences between PPV and sensitivity (even if one of these is very high), allowing diagnostic accuracy thresholds based on combined PPV and sensitivity to be determined. Therefore, CSI or F measures may be helpful complementary metrics to report alongside PPV and sensitivity in diagnostic accuracy studies of administrative epilepsy data.

A Simplified Correlation Index for Fast Real-Time Pulse Shape Recognition

A simplified correlation index is proposed to be used in real-time pulse shape recognition systems. This index is similar to the classic Pearson's correlation coefficient, but it can be efficiently implemented in FPGA devices with far fewer logic resources and excellent performance. Numerical simulations with synthetic data and comparisons with the Pearson's correlation show the suitability of the proposed index in applications such as the discrimination and counting of pulses with a predefined shape. Superior performance is evident in signal-to-noise ratio scenarios close to unity. FPGA implementation of Person's method and the proposed correlation index have been successfully tested and the main results are summarized.

Intracranial bruit: Charles Warlow's challenge revisited

Over 20 years ago, Charles Warlow, the founding editor of Practical Neurology, offered a copy of his stroke textbook to anyone diagnosing an intracranial arteriovenous malformation by auscultation of the skull alone. This article examines the possible diagnostic value of intracranial bruit in terms of the 2×2 contingency table for diagnostic tests and recounts an historical case.