Evaluating the Clinical Feasibility of an Artificial Intelligence-Powered, Web-Based Clinical Decision Support System for the Treatment of Depression in Adults: Longitudinal Feasibility Study

Clinical decision support systems in addiction and concurrent disorders: A systematic review and meta-analysis

INTRODUCTION

This review aims to synthesise the literature on the efficacy, evolution, and challenges of implementing Clincian Decision Support Systems (CDSS) in the realm of mental health, addiction, and concurrent disorders.

METHODS

Following PRISMA guidelines, a systematic review and meta-analysis were performed. Searches conducted in databases such as MEDLINE, Embase, CINAHL, PsycINFO, and Web of Science through 25 May 2023, yielded 27,344 records. After necessary exclusions, 69 records were allocated for detailed synthesis. In the examination of patient outcomes with a focus on metrics such as therapeutic efficacy, patient satisfaction, and treatment acceptance, meta-analytic techniques were employed to synthesise data from randomised controlled trials.

RESULTS

A total of 69 studies were included, revealing a shift from knowledge-based models pre-2017 to a rise in data-driven models post-2017. The majority of models were found to be in Stage 2 or 4 of maturity. The meta-analysis showed an effect size of -0.11 for addiction-related outcomes and a stronger effect size of -0.50 for patient satisfaction and acceptance of CDSS.

DISCUSSION

The results indicate a shift from knowledge-based to data-driven CDSS approaches, aligned with advances in machine learning and big data. Although the immediate impact on addiction outcomes is modest, higher patient satisfaction suggests promise for wider CDSS use. Identified challenges include alert fatigue and opaque AI models.

CONCLUSION

CDSS shows promise in mental health and addiction treatment but requires a nuanced approach for effective and ethical implementation. The results emphasise the need for continued research to ensure optimised and equitable use in healthcare settings.

Development of a differential treatment selection model for depression on consolidated and transformed clinical trial datasets

Major depressive disorder (MDD) is the leading cause of disability worldwide, yet treatment selection still proceeds via "trial and error". Given the varied presentation of MDD and heterogeneity of treatment response, the use of machine learning to understand complex, non-linear relationships in data may be key for treatment personalization. Well-organized, structured data from clinical trials with standardized outcome measures is useful for training machine learning models; however, combining data across trials poses numerous challenges. There is also persistent concern that machine learning models can propagate harmful biases. We have created a methodology for organizing and preprocessing depression clinical trial data such that transformed variables harmonized across disparate datasets can be used as input for feature selection. Using Bayesian optimization, we identified an optimal multi-layer dense neural network that used data from 21 clinical and sociodemographic features as input in order to perform differential treatment benefit prediction. With this combined dataset of 5032 individuals and 6 drugs, we created a differential treatment benefit prediction model. Our model generalized well to the held-out test set and produced similar accuracy metrics in the test and validation set with an AUC of 0.7 when predicting binary remission. To address the potential for bias propagation, we used a bias testing performance metric to evaluate the model for harmful biases related to ethnicity, age, or sex. We present a full pipeline from data preprocessing to model validation that was employed to create the first differential treatment benefit prediction model for MDD containing 6 treatment options.

Trust and acceptability of data-driven clinical recommendations in everyday practice: A scoping review

BACKGROUND

Increasing attention is being given to the analysis of large health datasets to derive new clinical decision support systems (CDSS). However, few data-driven CDSS are being adopted into clinical practice. Trust in these tools is believed to be fundamental for acceptance and uptake but to date little attention has been given to defining or evaluating trust in clinical settings.

OBJECTIVES

A scoping review was conducted to explore how and where acceptability and trustworthiness of data-driven CDSS have been assessed from the health professional's perspective.

METHODS

Medline, Embase, PsycInfo, Web of Science, Scopus, ACM Digital, IEEE Xplore and Google Scholar were searched in March 2022 using terms expanded from: "data-driven" AND "clinical decision support" AND "acceptability". Included studies focused on healthcare practitioner-facing data-driven CDSS, relating directly to clinical care. They included trust or a proxy as an outcome, or in the discussion. The preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (PRISMA-ScR) is followed in the reporting of this review.

RESULTS

3291 papers were screened, with 85 primary research studies eligible for inclusion. Studies covered a diverse range of clinical specialisms and intended contexts, but hypothetical systems (24) outnumbered those in clinical use (18). Twenty-five studies measured trust, via a wide variety of quantitative, qualitative and mixed methods. A further 24 discussed themes of trust without it being explicitly evaluated, and from these, themes of transparency, explainability, and supporting evidence were identified as factors influencing healthcare practitioner trust in data-driven CDSS.

CONCLUSION

There is a growing body of research on data-driven CDSS, but few studies have explored stakeholder perceptions in depth, with limited focused research on trustworthiness. Further research on healthcare practitioner acceptance, including requirements for transparency and explainability, should inform clinical implementation.

Towards Outcome-Driven Patient Subgroups: A Machine Learning Analysis Across Six Depression Treatment Studies

BACKGROUND

Major depressive disorder (MDD) is a heterogeneous condition; multiple underlying neurobiological and behavioral substrates are associated with treatment response variability. Understanding the sources of this variability and predicting outcomes has been elusive. Machine learning (ML) shows promise in predicting treatment response in MDD, but its application is limited by challenges to the clinical interpretability of ML models, and clinicians often lack confidence in model results. In order to improve the interpretability of ML models in clinical practice, our goal was to demonstrate the derivation of treatment-relevant patient profiles comprised of clinical and demographic information using a novel ML approach.

METHODS

We analyzed data from six clinical trials of pharmacological treatment for depression (total n = 5438) using the Differential Prototypes Neural Network (DPNN), a ML model that derives patient prototypes which can be used to derive treatment-relevant patient clusters while learning to generate probabilities for differential treatment response. A model classifying remission and outputting individual remission probabilities for five first-line monotherapies and three combination treatments was trained using clinical and demographic data. Prototypes were evaluated for interpretability by assessing differences in feature distributions (e.g. age, sex, symptom severity) and treatment-specific outcomes.

RESULTS

A 3-prototype model achieved an area under the receiver operating curve of 0.66 and an expected absolute improvement in remission rate for those receiving the best predicted treatment of 6.5% (relative improvement of 15.6%) compared to the population remission rate. We identified three treatment-relevant patient clusters. Cluster A patients tended to be younger, to have increased levels of fatigue, and more severe symptoms. Cluster B patients tended to be older, female, have less severe symptoms, and the highest remission rates. Cluster C patients had more severe symptoms, lower remission rates, more psychomotor agitation, more intense suicidal ideation, and more somatic genital symptoms.

CONCLUSION

It is possible to produce novel treatment-relevant patient profiles using ML models; doing so may improve interpretability of ML models and the quality of precision medicine treatments for MDD.

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

Machine learning (ML)-based risk prediction models hold the potential to support the health-care setting in several ways; however, use of such models is scarce. We aimed to review health-care professional (HCP) and patient perceptions of ML risk prediction models in published literature, to inform future risk prediction model development. Following database and citation searches, we identified 41 articles suitable for inclusion. Article quality varied with qualitative studies performing strongest. Overall, perceptions of ML risk prediction models were positive. HCPs and patients considered that models have the potential to add benefit in the health-care setting. However, reservations remain; for example, concerns regarding data quality for model development and fears of unintended consequences following ML model use. We identified that public views regarding these models might be more negative than HCPs and that concerns (eg, extra demands on workload) were not always borne out in practice. Conclusions are tempered by the low number of patient and public studies, the absence of participant ethnic diversity, and variation in article quality. We identified gaps in knowledge (particularly views from under-represented groups) and optimum methods for model explanation and alerts, which require future research.

Data-Driven and Technology-Enabled Trial Innovations Toward Decentralization of Clinical Trials: Opportunities and Considerations

Traditional trial designs have well-recognized inefficiencies and logistical barriers to participation. Decentralized trials and digital health solutions have been suggested as potential solutions and have certainly risen to the challenge during the pandemic. Clinical trial designs are now increasingly data driven. The use of distributed clinical data networks and digitization has helped to fundamentally upgrade existing research systems. A trial design may vary anywhere from fully decentralized to hybrid to traditional on-site. Various decentralization components are available for stakeholders to increase the reach and pace of their trials, such as electronic informed consent, remote interviews, administration, outcome assessment, monitoring, and laboratory and imaging modalities. Furthermore, digital health technologies can be included to enrich study conduct. However, careful consideration is warranted, including assessing verification and validity through usability studies and having various contingencies in place through dedicated risk assessment. Selecting the right combination depends not just on the ability to handle patient care and the medical know-how but also on the availability of appropriate technologic infrastructure, skills, and human resources. Throughout this process, quality of evidence generation and physician-patient relation must not be undermined. Here we also address some knowledge gaps, cost considerations, and potential impact of decentralization and digitization on inclusivity, recruitment, engagement, and retention. Last, we mention some future directions that may help drive the necessary change in the right direction.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

A personalized intervention to prevent depression in primary care based on risk predictive algorithms and decision support systems: protocol of the e-predictD study

The predictD is an intervention implemented by general practitioners (GPs) to prevent depression, which reduced the incidence of depression-anxiety and was cost-effective. The e-predictD study aims to design, develop, and evaluate an evolved predictD intervention to prevent the onset of major depression in primary care based on Information and Communication Technologies, predictive risk algorithms, decision support systems (DSSs), and personalized prevention plans (PPPs). A multicenter cluster randomized trial with GPs randomly assigned to the e-predictD intervention + care-as-usual (CAU) group or the active-control + CAU group and 1-year follow-up is being conducted. The required sample size is 720 non-depressed patients (aged 18-55 years), with moderate-to-high depression risk, under the care of 72 GPs in six Spanish cities. The GPs assigned to the e-predictD-intervention group receive brief training, and those assigned to the control group do not. Recruited patients of the GPs allocated to the e-predictD group download the e-predictD app, which incorporates validated risk algorithms to predict depression, monitoring systems, and DSSs. Integrating all inputs, the DSS automatically proposes to the patients a PPP for depression based on eight intervention modules: physical exercise, social relationships, improving sleep, problem-solving, communication skills, decision-making, assertiveness, and working with thoughts. This PPP is discussed in a 15-min semi-structured GP-patient interview. Patients then choose one or more of the intervention modules proposed by the DSS to be self-implemented over the next 3 months. This process will be reformulated at 3, 6, and 9 months but without the GP-patient interview. Recruited patients of the GPs allocated to the control-group+CAU download another version of the e-predictD app, but the only intervention that they receive via the app is weekly brief psychoeducational messages (active-control group). The primary outcome is the cumulative incidence of major depression measured by the Composite International Diagnostic Interview at 6 and 12 months. Other outcomes include depressive symptoms (PHQ-9) and anxiety symptoms (GAD-7), depression risk (predictD risk algorithm), mental and physical quality of life (SF-12), and acceptability and satisfaction ('e-Health Impact' questionnaire) with the intervention. Patients are evaluated at baseline and 3, 6, 9, and 12 months. An economic evaluation will also be performed (cost-effectiveness and cost-utility analysis) from two perspectives, societal and health systems.

Trial registration

ClinicalTrials.gov, identifier: NCT03990792.

Barriers and solutions to the adoption of translational tools for computational psychiatry

Computational psychiatry is a field aimed at developing formal models of information processing in the human brain, and how alterations in this processing can lead to clinical phenomena. There has been significant progress in the development of tasks and how to model them, presenting an opportunity to incorporate computational psychiatry methodologies into large- scale research projects or into clinical practice. In this viewpoint, we explore some of the barriers to incorporation of computational psychiatry tasks and models into wider mainstream research directions. These barriers include the time required for participants to complete tasks, test-retest reliability, limited ecological validity, as well as practical concerns, such as lack of computational expertise and the expense and large sample sizes traditionally required to validate tasks and models. We then discuss solutions, such as the redesigning of tasks with a view toward feasibility, and the integration of tasks into more ecologically valid and standardized game platforms that can be more easily disseminated. Finally, we provide an example of how one task, the conditioned hallucinations task, might be translated into such a game. It is our hope that interest in the creation of more accessible and feasible computational tasks will help computational methods make more positive impacts on research as well as, eventually, clinical practice.

Artificial intelligence (AI) and machine learning (ML) based decision support systems in mental health: An integrative review

An integrative review investigating the incorporation of artificial intelligence (AI) and machine learning (ML) based decision support systems in mental health care settings was undertaken of published literature between 2016 and 2021 across six databases. Four studies met the research question and the inclusion criteria. The primary theme identified was trust and confidence. To date, there is limited research regarding the use of AI-based decision support systems in mental health. Our review found that significant barriers exist regarding its incorporation into practice primarily arising from uncertainty related to clinician's trust and confidence, end-user acceptance and system transparency. More research is needed to understand the role of AI in assisting treatment and identifying missed care. Researchers and developers must focus on establishing trust and confidence with clinical staff before true clinical impact can be determined. Finally, further research is required to understand the attitudes and beliefs surrounding the use of AI and related impacts for the wellbeing of the end-users of care. This review highlights the necessity of involving clinicians in all stages of research, development and implementation of artificial intelligence in care delivery. Earning the trust and confidence of clinicians should be foremost in consideration in implementation of any AI-based decision support system. Clinicians should be motivated to actively embrace the opportunity to contribute to the development and implementation of new health technologies and digital tools that assist all health care professionals to identify missed care, before it occurs as a matter of importance for public safety and ethical implementation. AI-basesd decision support tools in mental health settings show most promise as trust and confidence of clinicians is achieved.

A systematic meta-review of patient-level predictors of psychological therapy outcome in major depressive disorder

BACKGROUND

Psychological therapies are effective for treating major depressive disorder, but current clinical guidelines do not provide guidance on the personalization of treatment choice. Established predictors of psychotherapy treatment response could help inform machine learning models aimed at predicting individual patient responses to different therapy options. Here we sought to comprehensively identify known predictors.

METHODS

EMBASE, Medline, PubMed, PsycINFO were searched for systematic reviews with or without meta-analysis published until June 2020 to identify individual patient-level predictors of response to psychological treatments. 3113 abstracts were identified and 300 articles assessed. We qualitatively synthesized our findings by predictor category (sociodemographic; symptom profile; social support; personality features; affective, cognitive, and behavioural; comorbidities; neuroimaging; genetics) and treatment type. We used the AMSTAR 2 to evaluate the quality of included reviews.

RESULTS

Following screening and full-text assessment, 27 systematic reviews including 12 meta-analyses were eligible for inclusion. 74 predictors emerged for various psychological treatments, primarily cognitive behavioural therapy, interpersonal therapy, and mindfulness-based cognitive therapy.

LIMITATIONS

A paucity of studies examining predictors of psychological treatment outcome, as well as methodological heterogeneities and publication biases limit the strength of the identified predictors.

CONCLUSIONS

The synthesized predictors could be used to supplement clinical decision-making in selecting psychological therapies based on individual patient characteristics. These predictors could also be used as a priori input features for machine learning models aimed at predicting a given patient's likelihood of response to different treatment options for depression, and may contribute toward the development of patient-specific treatment recommendations in clinical guidelines.