Accuracy versus reliability-based modelling approaches for medical decision making

A reliable jumping-based classification methodology for environment sector

Decision-makers have consistently developed a range of classification models, each possessing unique features within the domain of intelligent models. These endeavors are all directed toward achieving the highest levels of accuracy. In recent developments, two notable methodologies-reliable modeling and jumping modeling approaches-offer specific advantages in formulating cost functions and have been recognized for their role in enhancing classifier accuracy. Specifically, the jumping methodology is based on aligning the learning process with the discrete nature of the classification goal, while the reliable methodology integrates the reliability factor into the learning paradigm. However, their innovative combination, leveraging both accuracy and reliability factors in guiding learning processes, leads to the creation of a high-performing classifier. This addresses a research gap in tackling classification challenges, which remains the core focus of the present study. To evaluate the performance of the proposed reliable jumping-based intelligent classifier in environmental decision-making, we considered ten benchmark datasets spanning various application domains. The numerical results demonstrate that the proposed Reliable Jumping-based intelligent classifier consistently outperforms traditional intelligent classifiers across all studied cases. As a result, the proposed approach proves to be a viable and effective alternative to other intelligent methods in environmental applications.

Etemadi regression in chemometrics: Reliability-based procedures for modeling and forecasting

The creation of predictive models with a high degree of generalizability in chemical analysis and process optimization is of paramount importance. Nonetheless, formulating a prediction model based on collected data from chemical measurements that maximize quantitative generalizability remains a challenging task for chemometrics experts. To tackle this challenge, a range of forecasting models with varying characteristics, structures, and capabilities has been developed, utilizing either accuracy-based or reliability-based modeling strategies. While the majority of models follow the accuracy-based approach, a recently proposed reliability-based approach, known as the Etemadi approach, has shown impressive performance across various scientific fields. The Etemadi models were constructed through a reliability-based parameter estimation process in such a manner that maximizes the models' reliability. However, the foundation of modeling procedures for chemometrics purposes is built upon the assumption that high generalizability in inaccessible/test data is achieved through the accuracy-based training procedure in which errors in available historical/training data are minimized. After conducting a thorough review of the current literature, we have found that none of the forecasting models for chemometrics purposes incorporate reliability into their modeling procedures. Given the dynamic and highly sensitive nature of chemistry experiments and processes, implementing a reliable model that controls performance criteria variation is a promising strategy for achieving stable and robust forecasts. To address this research gap, this paper introduces several key innovations, which can be highlighted as follows: (1) Proposing a general design structure based on a new optimal reliability-based parameter estimation process. (2) Introducing a novel risk-based modeling strategy that minimizes the performance variation of models implemented under different conditions in chemical laboratory experiments, to generate a more generalizable model for diverse applications in chemometrics. (3) Specifying the degree of influence that each reliability and accuracy factor has in enhancing the generalizability and uncertainty modeling of chemometric models. Empirical evidence confirms the effectiveness and superior performance of reliability-based models compared to accuracy-based models in 78.95% of the cases across various fields, including Pharmacology, Biochemistry, Agrochemical, Geochemical, Biological, Pollutants, Physicochemical Properties, and Gases Experiment. Furthermore, the study's findings demonstrate that the reliability-based modeling approach outperforms the accuracy-based strategy in terms of MAE, MSE, ARV, and RMSE by an average of 4.697%, 5.646%, 5.646%, and 4.342%, respectively. It is also statistically proven that reliability has a more significant impact on improving the generalizability of chemometric models than accuracy. This emphasizes the importance of including reliability as a crucial factor in chemometrics modeling, a consideration that has been overlooked in traditional modeling processes. Consequently, reliability-based modeling approaches can be regarded as a viable alternative to conventional accuracy-based modeling methods for chemical modeling purposes.

Predicting outcomes at the individual patient level: what is the best method?

OBJECTIVE

When developing prediction models, researchers commonly employ a single model which uses all the available data (end-to-end approach). Alternatively, a similarity-based approach has been previously proposed, in which patients with similar clinical characteristics are first grouped into clusters, then prediction models are developed within each cluster. The potential advantage of the similarity-based approach is that it may better address heterogeneity in patient characteristics. However, it remains unclear whether it improves the overall predictive performance. We illustrate the similarity-based approach using data from people with depression and empirically compare its performance with the end-to-end approach.

METHODS

We used primary care data collected in general practices in the UK. Using 31 predefined baseline variables, we aimed to predict the severity of depressive symptoms, measured by Patient Health Questionnaire-9, 60 days after initiation of antidepressant treatment. Following the similarity-based approach, we used k-means to cluster patients based on their baseline characteristics. We derived the optimal number of clusters using the Silhouette coefficient. We used ridge regression to build prediction models in both approaches. To compare the models' performance, we calculated the mean absolute error (MAE) and the coefficient of determination (R2) using bootstrapping.

RESULTS

We analysed data from 16 384 patients. The end-to-end approach resulted in an MAE of 4.64 and R2 of 0.20. The best-performing similarity-based model was for four clusters, with MAE of 4.65 and R2 of 0.19.

CONCLUSIONS

The end-to-end and the similarity-based model yielded comparable performance. Due to its simplicity, the end-to-end approach can be favoured when using demographic and clinical data to build prediction models on pharmacological treatments for depression.