1
|
Mbizvo GK, Larner AJ. On the Dependence of the Critical Success Index (CSI) on Prevalence. Diagnostics (Basel) 2024; 14:545. [PMID: 38473017 DOI: 10.3390/diagnostics14050545] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2024] [Revised: 02/28/2024] [Accepted: 03/01/2024] [Indexed: 03/14/2024] Open
Abstract
The critical success index (CSI) is an established metric used in meteorology to verify the accuracy of weather forecasts. It is defined as the ratio of hits to the sum of hits, false alarms, and misses. Translationally, CSI has gained popularity as a unitary outcome measure in various clinical situations where large numbers of true negatives may influence the interpretation of other, more traditional, outcome measures, such as specificity (Spec) and negative predictive value (NPV), or when unified interpretation of positive predictive value (PPV) and sensitivity (Sens) is needed. The derivation of CSI from measures including PPV has prompted questions as to whether and how CSI values may vary with disease prevalence (P), just as PPV estimates are dependent on P, and hence whether CSI values are generalizable between studies with differing prevalences. As no detailed study of the relation of CSI to prevalence has been undertaken hitherto, the dataset of a previously published test accuracy study of a cognitive screening instrument was interrogated to address this question. Three different methods were used to examine the change in CSI across a range of prevalences, using both the Bayes formula and equations directly relating CSI to Sens, PPV, P, and the test threshold (Q). These approaches showed that, as expected, CSI does vary with prevalence, but the dependence differs according to the method of calculation that is adopted. Bayesian rescaling of both Sens and PPV generates a concave curve, suggesting that CSI will be maximal at a particular prevalence, which may vary according to the particular dataset.
Collapse
Affiliation(s)
- Gashirai K Mbizvo
- Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Biosciences Building, Crown Street, Liverpool L69 7BE, UK
- Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool L14 3PE, UK
- Cognitive Function Clinic, The Walton Centre NHS Foundation Trust, Liverpool L9 7LJ, UK
| | - Andrew J Larner
- Cognitive Function Clinic, The Walton Centre NHS Foundation Trust, Liverpool L9 7LJ, UK
| |
Collapse
|
2
|
Ribeiro P, Sá J, Paiva D, Rodrigues PM. Cardiovascular Diseases Diagnosis Using an ECG Multi-Band Non-Linear Machine Learning Framework Analysis. Bioengineering (Basel) 2024; 11:58. [PMID: 38247935 PMCID: PMC10813154 DOI: 10.3390/bioengineering11010058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Revised: 12/13/2023] [Accepted: 01/05/2024] [Indexed: 01/23/2024] Open
Abstract
BACKGROUND cardiovascular diseases (CVDs), which encompass heart and blood vessel issues, stand as the leading cause of global mortality for many people. METHODS the present study intends to perform discrimination between seven well-known CVDs (bundle branch block, cardiomyopathy, myocarditis, myocardial hypertrophy, myocardial infarction, valvular heart disease, and dysrhythmia) and one healthy control group, respectively, by feeding a set of machine learning (ML) models with 10 non-linear features extracted every 1 s from electrocardiography (ECG) lead signals of a well-known ECG database (PTB diagnostic ECG database) using multi-band analysis performed by discrete wavelet transform (DWT). The ML models were trained and tested using a leave-one-out cross-validation approach, assessing the individual and combined capabilities of features, per each lead or combined, to distinguish between pairs of study groups and for conducting a comprehensive all vs. all analysis. RESULTS the Accuracy discrimination results ranged between 73% and 100%, the Recall between 68% and 100%, and the AUC between 0.42 and 1. CONCLUSIONS the results suggest that our method is a good tool for distinguishing CVDs, offering significant advantages over other studies that used the same dataset, including a multi-class comparison group (all vs. all), a wider range of binary comparisons, and the use of classical non-linear analysis under ECG multi-band analysis performed by DWT.
Collapse
Affiliation(s)
| | | | | | - Pedro Miguel Rodrigues
- CBQF—Centro de Biotecnologia e Química Fina, Laboratório Associado, Escola Superior de Biotecnologia, Universidade Católica Portuguesa, Rua de Diogo Botelho 1327, 4169-005 Porto, Portugal; (P.R.); (J.S.); (D.P.)
| |
Collapse
|
3
|
Mbizvo GK, Simpson CR, Duncan SE, Chin RFM, Larner AJ. Critical success index or F measure to validate the accuracy of administrative healthcare data identifying epilepsy in deceased adults in Scotland. Epilepsy Res 2024; 199:107275. [PMID: 38128202 DOI: 10.1016/j.eplepsyres.2023.107275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Revised: 11/20/2023] [Accepted: 12/05/2023] [Indexed: 12/23/2023]
Abstract
BACKGROUND Methods to undertake diagnostic accuracy studies of administrative epilepsy data are challenged by lack of a way to reliably rank case-ascertainment algorithms in order of their accuracy. This is because it is difficult to know how to prioritise positive predictive value (PPV) and sensitivity (Sens). Large numbers of true negative (TN) instances frequently found in epilepsy studies make it difficult to discriminate algorithm accuracy on the basis of negative predictive value (NPV) and specificity (Spec) as these become inflated (usually >90%). This study demonstrates the complementary value of using weather forecasting or machine learning metrics critical success index (CSI) or F measure, respectively, as unitary metrics combining PPV and sensitivity. We reanalyse data published in a diagnostic accuracy study of administrative epilepsy mortality data in Scotland. METHOD CSI was calculated as 1/[(1/PPV) + (1/Sens) - 1]. F measure was calculated as 2.PPV.Sens/(PPV + Sens). CSI and F values range from 0 to 1, interpreted as 0 = inaccurate prediction and 1 = perfect accuracy. The published algorithms were reanalysed using these and their accuracy re-ranked according to CSI in order to allow comparison to the original rankings. RESULTS CSI scores were conservative (range 0.02-0.826), always less than or equal to the lower of the corresponding PPV (range 39-100%) and sensitivity (range 2-93%). F values were less conservative (range 0.039-0.905), sometimes higher than either PPV or sensitivity, but were always higher than CSI. Low CSI and F values occurred when there was a large difference between PPV and sensitivity, e.g. CSI was 0.02 and F was 0.039 in an instance when PPV was 100% and sensitivity was 2%. Algorithms with both high PPV and sensitivity performed best in terms of CSI and F measure, e.g. CSI was 0.826 and F was 0.905 in an instance when PPV was 90% and sensitivity was 91%. CONCLUSION CSI or F measure can combine PPV and sensitivity values into a convenient single metric that is easier to interpret and rank in terms of diagnostic accuracy than trying to rank diagnostic accuracy according to the two measures themselves. CSI or F prioritise instances where both PPV and sensitivity are high over instances where there are large differences between PPV and sensitivity (even if one of these is very high), allowing diagnostic accuracy thresholds based on combined PPV and sensitivity to be determined. Therefore, CSI or F measures may be helpful complementary metrics to report alongside PPV and sensitivity in diagnostic accuracy studies of administrative epilepsy data.
Collapse
Affiliation(s)
- Gashirai K Mbizvo
- Pharmacology and Therapeutics, Institute of Systems, Molecular and Integrative Biology, University of Liverpool, Liverpool, United Kingdom; Liverpool Centre of Cardiovascular Science at University of Liverpool, Liverpool John Moores University and Liverpool Heart and Chest Hospital, Liverpool, United Kingdom; Cognitive Function Clinic, Walton Centre NHS Foundation Trust, Liverpool, United Kingdom.
| | - Colin R Simpson
- School of Health, Wellington Faculty of Health, Victoria University of Wellington, Wellington, New Zealand; The Usher Institute, The University of Edinburgh, Edinburgh, United Kingdom.
| | - Susan E Duncan
- Muir Maxwell Epilepsy Centre, Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom; Department of Clinical Neurosciences, NHS Lothian, Edinburgh, United Kingdom.
| | - Richard F M Chin
- Muir Maxwell Epilepsy Centre, Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, United Kingdom; Royal Hospital for Children and Young People, Edinburgh, United Kingdom.
| | - Andrew J Larner
- Cognitive Function Clinic, Walton Centre NHS Foundation Trust, Liverpool, United Kingdom.
| |
Collapse
|
4
|
Cicuttin A, Morales IR, Crespo ML, Carrato S, García LG, Molina RS, Valinoti B, Folla Kamdem J. A Simplified Correlation Index for Fast Real-Time Pulse Shape Recognition. SENSORS (BASEL, SWITZERLAND) 2022; 22:7697. [PMID: 36298048 PMCID: PMC9607046 DOI: 10.3390/s22207697] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 10/01/2022] [Accepted: 10/04/2022] [Indexed: 06/16/2023]
Abstract
A simplified correlation index is proposed to be used in real-time pulse shape recognition systems. This index is similar to the classic Pearson's correlation coefficient, but it can be efficiently implemented in FPGA devices with far fewer logic resources and excellent performance. Numerical simulations with synthetic data and comparisons with the Pearson's correlation show the suitability of the proposed index in applications such as the discrimination and counting of pulses with a predefined shape. Superior performance is evident in signal-to-noise ratio scenarios close to unity. FPGA implementation of Person's method and the proposed correlation index have been successfully tested and the main results are summarized.
Collapse
Affiliation(s)
- Andres Cicuttin
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
| | - Iván René Morales
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
- Dipartimento di Ingegneria e Architettura, Università degli Studi di Trieste (UNITS), 34127 Trieste, Italy
| | - Maria Liz Crespo
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
| | - Sergio Carrato
- Dipartimento di Ingegneria e Architettura, Università degli Studi di Trieste (UNITS), 34127 Trieste, Italy
| | - Luis Guillermo García
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
| | - Romina Soledad Molina
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
- Dipartimento di Ingegneria e Architettura, Università degli Studi di Trieste (UNITS), 34127 Trieste, Italy
| | - Bruno Valinoti
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
- Dipartimento di Ingegneria e Architettura, Università degli Studi di Trieste (UNITS), 34127 Trieste, Italy
| | - Jerome Folla Kamdem
- Multidisciplinary Laboratory, The Abdus Salam International Centre for Theoretical Physics (ICTP), 34151 Trieste, Italy
- Department of Physics, University of Yaoundé I, P.O. Box 812, Yaoundé 222, Cameroon
| |
Collapse
|
5
|
Mbizvo GK, Larner AJ. Isolated headache is not a reliable indicator for brain cancer. Clin Med (Lond) 2022; 22:92-93. [PMID: 38589111 DOI: 10.7861/clinmed.let.22.1.2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Affiliation(s)
- Gashirai K Mbizvo
- Walton Centre for Neurology and Neurosurgery, Liverpool, UK; Walton Centre for Neurology and Neurosurgery, Liverpool, UK
| | - Andrew J Larner
- Walton Centre for Neurology and Neurosurgery, Liverpool, UK; Walton Centre for Neurology and Neurosurgery, Liverpool, UK
| |
Collapse
|
6
|
Larner AJ. Intracranial bruit: Charles Warlow's challenge revisited. Pract Neurol 2021; 22:79-81. [PMID: 34853127 DOI: 10.1136/practneurol-2021-003226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/20/2021] [Indexed: 11/03/2022]
Abstract
Over 20 years ago, Charles Warlow, the founding editor of Practical Neurology, offered a copy of his stroke textbook to anyone diagnosing an intracranial arteriovenous malformation by auscultation of the skull alone. This article examines the possible diagnostic value of intracranial bruit in terms of the 2×2 contingency table for diagnostic tests and recounts an historical case.
Collapse
Affiliation(s)
- Andrew J Larner
- Cognitive Function Clinic, Walton Centre for Neurology and Neurosurgery, Liverpool, UK
| |
Collapse
|
7
|
Burns A, Larner AJ. Do we need yet another cognitive test? Free-Cog, a novel, hybrid, cognitive screening instrument. J Neurol Neurosurg Psychiatry 2021; 92:1359-1360. [PMID: 33722823 DOI: 10.1136/jnnp-2020-325830] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/07/2020] [Accepted: 02/10/2021] [Indexed: 11/04/2022]
Affiliation(s)
- Alistair Burns
- Department of Psychiatry and Behavioural Sciences, University of Manchester, Manchester, UK
| | - A J Larner
- Cognitive Function Clinic, Walton Centre for Neurology and Neurosurgery, Liverpool, Liverpool, UK
| |
Collapse
|