Development and validation of clinical prediction models: marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming

Risk calculators for the detection of prostate cancer: a systematic review

BACKGROUND

Prostate cancer (PCa) (early) detection poses significant challenges, including unnecessary testing and the risk of potential overdiagnosis. The European Association of Urology therefore suggests an individual risk-adapted approach, incorporating risk calculators (RCs) into the PCa detection pathway. In the context of 'The PRostate Cancer Awareness and Initiative for Screening in the European Union' (PRAISE-U) project ( https://uroweb.org/praise-u ), we aim to provide an overview of the currently available clinical RCs applicable in an early PCa detection algorithm.

METHODS

We performed a systematic review to identify RCs predicting detection of clinically significant PCa at biopsy. A search was performed in the databases Medline ALL, Embase, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar for publications between January 2010 and July 2023. We retrieved relevant literature by using the terms "prostate cancer", "screening/diagnosis" and "predictive model". Inclusion criteria included systematic reviews, meta-analyses, and clinical trials. Exclusion criteria applied to studies involving pre-targeted high-risk populations, diagnosed PCa patients, or a sample sizes under 50 men.

RESULTS

We identified 6474 articles, of which 140 were included after screening abstracts and full texts. In total, we identified 96 unique RCs. Among these, 45 underwent external validation, with 28 validated in multiple cohorts. Of the externally validated RCs, 17 are based on clinical factors, 19 incorporate clinical factors along with MRI details, 4 were based on blood biomarkers alone or in combination with clinical factors, and 5 included urinary biomarkers. The median AUC of externally validated RCs ranged from 0.63 to 0.93.

CONCLUSIONS

This systematic review offers an extensive analysis of currently available RCs, their variable utilization, and performance within validation cohorts. RCs have consistently demonstrated their capacity to mitigate the limitations associated with early detection and have been integrated into modern practice and screening trials. Nevertheless, the lack of external validation data raises concerns about numerous RCs, and it is crucial to factor in this omission when evaluating whether a specific RC is applicable to one's target population.

Lymph node metastases and recurrence in pT1 colorectal cancer: Prediction with the International Budding Consortium Score-A retrospective, multi-centric study

BACKGROUND

The International Collaboration on Cancer Reporting proposes histological tumour type, lymphovascular invasion, tumour grade, perineural invasion, extent, and dimensions of invasion as risk factors for lymph node metastases and tumour progression in completely endoscopically resected pT1 colorectal cancer (CRC).

OBJECTIVE

The aim of the study was to propose a predictive and reliable score to optimise the clinical management of endoscopically resected pT1 CRC patients.

METHODS

This multi-centric, retrospective International Budding Consortium (IBC) study included an international pT1 CRC cohort of 565 patients. All cases were reviewed by eight expert gastrointestinal pathologists. All risk factors were reported according to international guidelines. Tumour budding and immune response (CD8+ T-cells) were assessed with automated models using artificial intelligence. We used the information on risk factors and least absolute shrinkage and selection operator logistic regression to develop a prediction model and generate a score to predict the occurrence of lymph node metastasis or cancer recurrence.

RESULTS

The IBC prediction score included the following parameters: lymphovascular invasion, tumour buds, infiltration depth and tumour grade. The score has an acceptable discrimination power (area under the curve of 0.68 [95% confidence intervals (CI) 0.61-0.75]; 0.64 [95% CI 0.57-0.71] after internal validation). At a cut-off of 6.8 points to discriminate high-and low-risk patients, the score had a sensitivity and specificity of 0.9 [95% CI 0.8-0.95] and 0.26 [95% 0.22, 0.3], respectively.

CONCLUSION

The IBC score is based on well-established risk factors and is a promising tool with clinical utility to support the management of pT1 CRC patients.

Application of Artificial Intelligence in Community-Based Primary Health Care: Systematic Scoping Review and Critical Appraisal

BACKGROUND

Research on the integration of artificial intelligence (AI) into community-based primary health care (CBPHC) has highlighted several advantages and disadvantages in practice regarding, for example, facilitating diagnosis and disease management, as well as doubts concerning the unintended harmful effects of this integration. However, there is a lack of evidence about a comprehensive knowledge synthesis that could shed light on AI systems tested or implemented in CBPHC.

OBJECTIVE

We intended to identify and evaluate published studies that have tested or implemented AI in CBPHC settings.

METHODS

We conducted a systematic scoping review informed by an earlier study and the Joanna Briggs Institute (JBI) scoping review framework and reported the findings according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Scoping Reviews) reporting guidelines. An information specialist performed a comprehensive search from the date of inception until February 2020, in seven bibliographic databases: Cochrane Library, MEDLINE, EMBASE, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), ScienceDirect, and IEEE Xplore. The selected studies considered all populations who provide and receive care in CBPHC settings, AI interventions that had been implemented, tested, or both, and assessed outcomes related to patients, health care providers, or CBPHC systems. Risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Two authors independently screened the titles and abstracts of the identified records, read the selected full texts, and extracted data from the included studies using a validated extraction form. Disagreements were resolved by consensus, and if this was not possible, the opinion of a third reviewer was sought. A third reviewer also validated all the extracted data.

RESULTS

We retrieved 22,113 documents. After the removal of duplicates, 16,870 documents were screened, and 90 peer-reviewed publications met our inclusion criteria. Machine learning (ML) (41/90, 45%), natural language processing (NLP) (24/90, 27%), and expert systems (17/90, 19%) were the most commonly studied AI interventions. These were primarily implemented for diagnosis, detection, or surveillance purposes. Neural networks (ie, convolutional neural networks and abductive networks) demonstrated the highest accuracy, considering the given database for the given clinical task. The risk of bias in diagnosis or prognosis studies was the lowest in the participant category (4/49, 4%) and the highest in the outcome category (22/49, 45%).

CONCLUSIONS

We observed variabilities in reporting the participants, types of AI methods, analyses, and outcomes, and highlighted the large gap in the effective development and implementation of AI in CBPHC. Further studies are needed to efficiently guide the development and implementation of AI interventions in CBPHC settings.

Accuracy of identifying hospital acquired venous thromboembolism by administrative coding: implications for big data and machine learning research

Big data analytics research using heterogeneous electronic health record (EHR) data requires accurate identification of disease phenotype cases and controls. Overreliance on ground truth determination based on administrative data can lead to biased and inaccurate findings. Hospital-acquired venous thromboembolism (HA-VTE) is challenging to identify due to its temporal evolution and variable EHR documentation. To establish ground truth for machine learning modeling, we compared accuracy of HA-VTE diagnoses made by administrative coding to manual review of gold standard diagnostic test results. We performed retrospective analysis of EHR data on 3680 adult stepdown unit patients identifying HA-VTE. International Classification of Diseases, Ninth Revision (ICD-9-CM) codes for VTE were identified. 4544 radiology reports associated with VTE diagnostic tests were screened using terminology extraction and then manually reviewed by a clinical expert to confirm diagnosis. Of 415 cases with ICD-9-CM codes for VTE, 219 were identified with acute onset type codes. Test report review identified 158 new-onset HA-VTE cases. Only 40% of ICD-9-CM coded cases (n = 87) were confirmed by a positive diagnostic test report, leaving the majority of administratively coded cases unsubstantiated by confirmatory diagnostic test. Additionally, 45% of diagnostic test confirmed HA-VTE cases lacked corresponding ICD codes. ICD-9-CM coding missed diagnostic test-confirmed HA-VTE cases and inaccurately assigned cases without confirmed VTE, suggesting dependence on administrative coding leads to inaccurate HA-VTE phenotyping. Alternative methods to develop more sensitive and specific VTE phenotype solutions portable across EHR vendor data are needed to support case-finding in big-data analytics.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

A genetic programming approach to development of clinical prediction models: A case study in symptomatic cardiovascular disease

Background

Genetic programming (GP) is an evolutionary computing methodology capable of identifying complex, non-linear patterns in large data sets. Despite the potential advantages of GP over more typical, frequentist statistical approach methods, its applications to survival analyses are rare, at best. The aim of this study was to determine the utility of GP for the automatic development of clinical prediction models.

Methods

We compared GP against the commonly used Cox regression technique in terms of the development and performance of a cardiovascular risk score using data from the SMART study, a prospective cohort study of patients with symptomatic cardiovascular disease. The composite endpoint was cardiovascular death, non-fatal stroke, and myocardial infarction. A total of 3,873 patients aged 19–82 years were enrolled in the study 1996–2006. The cohort was split 70:30 into derivation and validation sets. The derivation set was used for development of both GP and Cox regression models. These models were then used to predict the discrete hazards at t = 1, 3, and 5 years. The predictive ability of both models was evaluated in terms of their risk discrimination and calibration using the validation set.

Results

The discrimination of both models was comparable. At time points t = 1, 3, and 5 years the C-index was 0.59, 0.69, 0.64 and 0.66, 0.70, 0.70 for the GP and Cox regression models respectively. At the same time points, the calibration of both models, which was assessed using calibration plots and the generalization of the Hosmer-Lemeshow test statistic, was also comparable, but with the Cox model being better calibrated to the validation data.

Conclusion

Using empirical data, we demonstrated that a prediction model developed automatically by GP has predictive ability comparable to that of manually tuned Cox regression. The GP model was more complex, but it was developed in a fully automated way and comprised fewer covariates. Furthermore, it did not require the expertise normally needed for its derivation, thereby alleviating the knowledge elicitation bottleneck. Overall, GP demonstrated considerable potential as a method for the automated development of clinical prediction models for diagnostic and prognostic purposes.

Systematic review of risk prediction scores for surgical site infection or periprosthetic joint infection following joint arthroplasty

Accurate identification of individuals at high risk of surgical site infections (SSIs) or periprosthetic joint infections (PJIs) influences clinical decisions and development of preventive strategies. We aimed to determine progress in the development and validation of risk prediction models for SSI or PJI using a systematic review. We searched for studies that have developed or validated a risk prediction tool for SSI or PJI following joint replacement in MEDLINE, EMBASE, Web of Science and Cochrane databases; trial registers and reference lists of studies up to September 2016. Nine studies describing 16 risk scores for SSI or PJI were identified. The number of component variables in a risk score ranged from 4 to 45. The C-index ranged from 0·56 to 0·74, with only three risk scores reporting a discriminative ability of >0·70. Five risk scores were validated internally. The National Healthcare Safety Network SSIs risk models for hip and knee arthroplasties (HPRO and KPRO) were the only scores to be externally validated. Except for HPRO which shows some promise for use in a clinical setting (based on predictive performance and external validation), none of the identified risk scores can be considered ready for use. Further research is urgently warranted within the field.

The ecological context of soundscapes for children's blood pressure

Although the majority of studies on community noise levels and children's physiological stress responses are positive, effect sizes vary considerably, and some studies do not confirm these effects. Employing a contextual perspective congruent with soundscapes, a carefully constructed sample of children (N = 115, M = 10.1 yr) living in households in relatively high (>60 dBA) or low (<50 dBA) noise areas created by proximity to major traffic arterials in Austria was reanalyzed. Several personal and environmental factors known to affect resting cardiovascular parameters measured under well-controlled, clinical conditions were incorporated into the analyses. Children with premature births and elevated chronic stress (i.e., overnight cortisol) were more susceptible to adverse blood pressure responses to road traffic noise. Residence in a multi-dwelling unit as well as standardized assessments of perceived quietness of the area did not modify the traffic noise impacts but each had its own, independent effect on resting blood pressure. A primary air pollutant associated with traffic volume (NO2) had no influence on any of these results. The scope of environmental noise assessment and management would benefit from incorporation of a more contextualized approach as suggested by the soundscape perspective.

Validating and updating a risk model for pneumonia - a case study

Background

The development of risk prediction models is of increasing importance in medical research - their use in practice, however, is rare. Among other reasons this might be due to the fact that thorough validation is often lacking. This study focuses on two Bayesian approaches of how to validate a prediction rule for the diagnosis of pneumonia, and compares them with established validation methods.

Methods

Expert knowledge was used to derive a risk prediction model for pneumonia. Data on more than 600 patients presenting with cough and fever at a general practitioner’s practice in Switzerland were collected in order to validate the expert model and to examine the predictive performance of it. Additionally, four modifications of the original model including shrinkage of the regression coefficients, and two Bayesian approaches with the expert model used as prior mean and different weights for the prior covariance matrix were fitted. We quantify the predictive performance of the different methods with respect to calibration and discrimination, using cross-validation.

Results

The predictive performance of the unshrinked regression coefficients was poor when applied to the Swiss cohort. Shrinkage improved the results, but a Bayesian model formulation with unspecified weight of the informative prior lead to large AUC and small Brier score, naïve and after cross-validation. The advantage of this approach is the flexibility in case of a prior-data conflict.

Conclusions

Published risk prediction rules in clinical research need to be validated externally before they can be used in new settings. We propose to use a Bayesian model formulation with the original risk prediction rule as prior. The posterior means of the coefficients, given the validation data showed best predictive performance with respect to cross-validated calibration and discriminative ability.