Palliative Care in the Twenty-First Century: Using Advanced Analytics to Uncloak Insights from Big Data

Identification of Digital Health Priorities for Palliative Care Research: Modified Delphi Study

BACKGROUND

Developments in digital health have the potential to transform the delivery of health and social care to help citizens manage their health. Currently, there is a lack of consensus about digital health research priorities in palliative care and a lack of theories about how these technologies might improve care outcomes. Therefore, it is important for health care leaders to identify innovations to ensure that an increasingly frail population has appropriate access to palliative care services. Consequently, it is important to articulate research priorities as the first step in determining how finite resources should be allocated to a field saturated with rapidly developing innovation.

OBJECTIVE

The aim of this study is to identify research priority areas for digital health in palliative care.

METHODS

We selected digital health trends, most relevant to palliative care, from a list of emerging trends reported by a leading institute of quantitative futurists. We conducted 2 rounds of the Delphi questionnaire, followed by a consensus meeting and public engagement workshop to establish a final consensus on research priorities for digital technology in palliative care. We used the views of public representatives to gain their perspectives on the agreed priorities.

RESULTS

A total of 103 experts (representing 11 countries) participated in the first Delphi round. Of the 103 experts, 55 (53.3%) participated in the second round. The final consensus meetings were attended by 10.7% (11/103) of the experts. We identified 16 priority areas, which involved many applications of technologies, including care for patients and caregivers, self-management and reporting of diseases, education and training, communication, care coordination, and research methodology. We summarized the priority areas into eight topics: big data, mobile devices, telehealth and telemedicine, virtual reality, artificial intelligence, smart home, biotechnology, and digital legacy.

CONCLUSIONS

The priorities identified in this study represent a wide range of important emerging areas in the fields of digital health, personalized medicine, and data science. Human-centered design and robust governance systems should be considered in future research. It is important that the risks of using these technologies in palliative care are properly addressed to ensure that these tools are used meaningfully, wisely, and safely and do not cause unintentional harm.

Development, implementation, and prospective validation of a model to predict 60-day end-of-life in hospitalized adults upon admission at three sites

BACKGROUND

Automated systems that use machine learning to estimate a patient's risk of death are being developed to influence care. There remains sparse transparent reporting of model generalizability in different subpopulations especially for implemented systems.

METHODS

A prognostic study included adult admissions at a multi-site, academic medical center between 2015 and 2017. A predictive model for all-cause mortality (including initiation of hospice care) within 60 days of admission was developed. Model generalizability is assessed in temporal validation in the context of potential demographic bias. A subsequent prospective cohort study was conducted at the same sites between October 2018 and June 2019. Model performance during prospective validation was quantified with areas under the receiver operating characteristic and precision recall curves stratified by site. Prospective results include timeliness, positive predictive value, and the number of actionable predictions.

RESULTS

Three years of development data included 128,941 inpatient admissions (94,733 unique patients) across sites where patients are mostly white (61%) and female (60%) and 4.2% led to death within 60 days. A random forest model incorporating 9614 predictors produced areas under the receiver operating characteristic and precision recall curves of 87.2 (95% CI, 86.1-88.2) and 28.0 (95% CI, 25.0-31.0) in temporal validation. Performance marginally diverges within sites as the patient mix shifts from development to validation (patients of one site increases from 10 to 38%). Applied prospectively for nine months, 41,728 predictions were generated in real-time (median [IQR], 1.3 [0.9, 32] minutes). An operating criterion of 75% positive predictive value identified 104 predictions at very high risk (0.25%) where 65% (50 from 77 well-timed predictions) led to death within 60 days.

CONCLUSION

Temporal validation demonstrates good model discrimination for 60-day mortality. Slight performance variations are observed across demographic subpopulations. The model was implemented prospectively and successfully produced meaningful estimates of risk within minutes of admission.

Estimating real-world performance of a predictive model: a case-study in predicting mortality

OBJECTIVE

One primary consideration when developing predictive models is downstream effects on future model performance. We conduct experiments to quantify the effects of experimental design choices, namely cohort selection and internal validation methods, on (estimated) real-world model performance.

MATERIALS AND METHODS

Four years of hospitalizations are used to develop a 1-year mortality prediction model (composite of death or initiation of hospice care). Two common methods to select appropriate patient visits from their encounter history (backwards-from-outcome and forwards-from-admission) are combined with 2 testing cohorts (random and temporal validation). Two models are trained under otherwise identical conditions, and their performances compared. Operating thresholds are selected in each test set and applied to a "real-world" cohort of labeled admissions from another, unused year.

RESULTS

Backwards-from-outcome cohort selection retains 25% of candidate admissions (n = 23 579), whereas forwards-from-admission selection includes many more (n = 92 148). Both selection methods produce similar performances when applied to a random test set. However, when applied to the temporally defined "real-world" set, forwards-from-admission yields higher areas under the ROC and precision recall curves (88.3% and 56.5% vs. 83.2% and 41.6%).

DISCUSSION

A backwards-from-outcome experiment manipulates raw training data, simplifying the experiment. This manipulated data no longer resembles real-world data, resulting in optimistic estimates of test set performance, especially at high precision. In contrast, a forwards-from-admission experiment with a temporally separated test set consistently and conservatively estimates real-world performance.

CONCLUSION

Experimental design choices impose bias upon selected cohorts. A forwards-from-admission experiment, validated temporally, can conservatively estimate real-world performance.

LAY SUMMARY

The routine care of patients stands to benefit greatly from assistive technologies, including data-driven risk assessment. Already, many different machine learning and artificial intelligence applications are being developed from complex electronic health record data. To overcome challenges that arise from such data, researchers often start with simple experimental approaches to test their work. One key component is how patients (and their healthcare visits) are selected for the study from the pool of all patients seen. Another is how the group of patients used to create the risk estimator differs from the group used to evaluate how well it works. These choices complicate how the experimental setting compares to the real-world application to patients. For example, different selection approaches that depend on each patient's future outcome can simplify the experiment but are impractical upon implementation as these data are unavailable. We show that this kind of "backwards" experiment optimistically estimates how well the model performs. Instead, our results advocate for experiments that select patients in a "forwards" manner and "temporal" validation that approximates training on past data and implementing on future data. More robust results help gauge the clinical utility of recent works and aid decision-making before implementation into practice.