A Comparison of Different Modelling Techniques in Predicting Mortality with the Tilburg Frailty Indicator (Preprint)

External Validation of Models for Predicting Disability in Community-Dwelling Older People in the Netherlands: A Comparative Study

Background

Advanced statistical modeling techniques may help predict health outcomes. However, it is not the case that these modeling techniques always outperform traditional techniques such as regression techniques. In this study, external validation was carried out for five modeling strategies for the prediction of the disability of community-dwelling older people in the Netherlands.

Methods

We analyzed data from five studies consisting of community-dwelling older people in the Netherlands. For the prediction of the total disability score as measured with the Groningen Activity Restriction Scale (GARS), we used fourteen predictors as measured with the Tilburg Frailty Indicator (TFI). Both the TFI and the GARS are self-report questionnaires. For the modeling, five statistical modeling techniques were evaluated: general linear model (GLM), support vector machine (SVM), neural net (NN), recursive partitioning (RP), and random forest (RF). Each model was developed on one of the five data sets and then applied to each of the four remaining data sets. We assessed the performance of the models with calibration characteristics, the correlation coefficient, and the root of the mean squared error.

Results

The models GLM, SVM, RP, and RF showed satisfactory performance characteristics when validated on the validation data sets. All models showed poor performance characteristics for the deviating data set both for development and validation due to the deviating baseline characteristics compared to those of the other data sets.

Conclusion

The performance of four models (GLM, SVM, RP, RF) on the development data sets was satisfactory. This was also the case for the validation data sets, except when these models were developed on the deviating data set. The NN models showed a much worse performance on the validation data sets than on the development data sets.

Bayesian Techniques in Predicting Frailty among Community-Dwelling Older Adults in the Netherlands

Background Frailty is a syndrome that is defined as an accumulation of deficits in physical, psychological, and social domains. On a global scale, there is an urgent need to create frailty-ready healthcare systems due to the healthcare burden that frailty confers on systems and the increased risk of falls, healthcare utilization, disability, and premature mortality. Several studies have been conducted to develop prediction models for predicting frailty. Most studies used logistic regression as a technique to develop a prediction model. One area that has experienced significant growth is the application of Bayesian techniques, partly due to an increasing number of practitioners valuing the Bayesian paradigm as matching that of scientific discovery. Objective We compared ten different Bayesian networks as proposed by ten experts in the field of frail elderly people to predict frailty with a choice from ten dichotomized determinants for frailty. Methods We used the opinion of ten experts who could indicate, using an empty Bayesian network graph, the important predictors for frailty and the interactions between the different predictors. The candidate predictors were age, sex, marital status, ethnicity, education, income, lifestyle, multimorbidity, life events, and home living environment. The ten Bayesian network models were evaluated in terms of their ability to predict frailty. For the evaluation, we used the data of 479 participants that filled in the Tilburg Frailty indicator (TFI) questionnaire for assessing frailty among community-dwelling older people. The data set contained the aforementioned variables and the outcome "frail". The model fit of each model was measured using the Akaike information criterion (AIC) and the predictive performance of the models was measured using the area under the curve (AUC) of the receiver operator characteristic (ROC). The AUCs of the models were validated using bootstrapping with 100 repetitions. The relative importance of the predictors in the models was calculated using the permutation feature importance algorithm (PFI). Results The ten Bayesian networks of the ten experts differed considerably regarding the predictors and the connections between the predictors and the outcome. However, all ten networks had corrected AUCs >0.700. Evaluating the importance of the predictors in each model, "diseases or chronic disorders" was the most important predictor in all models (10 times). The predictors "lifestyle" and "monthly income" were also often present in the models (both 6 times). One or more diseases or chronic disorders, an unhealthy lifestyle, and a monthly income below 1800 euro increased the likelihood of frailty. Conclusions Although the ten experts all made different graphs, the predictive performance was always satisfying (AUCs >0.700). While it is true that the predictor importance varied all the time, the top three of the predictor importance consisted of "diseases or chronic disorders", "lifestyle" and "monthly income". All in all, asking for the opinion of experts in the field of frail elderly to predict frailty with Bayesian networks may be more rewarding than a data-driven forecast with Bayesian networks because they have expert knowledge regarding interactions between the different predictors.