A Framework for Evaluating Dashboards in Healthcare

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

BACKGROUND

CARA set out to develop a data-visualisation platform to facilitate general practitioners to develop a deeper understanding of their patient population, disease management and prescribing through dashboards. To support the continued use and sustainability of the CARA dashboards, dashboard performance and user engagement have to be optimised. User research places people at the centre of the design process and aims to evaluate the needs, behaviours and attitudes of users to inform the design, development and impact of a product.

OBJECTIVE

To explore how different initial key messages impact the level of behavioural engagement with a CARA dashboard.

METHODS

Participating general practices can upload their practice data for analysis and visualisation in CARA dashboards. Practices will be randomised to one of three different initial landing pages: the full dashboard or one of two key messages: a between comparison (their practice prescribing with the average of all other practices) or within comparison (with practice data of the same month the previous year) with subsequent continuation to the full dashboard. Analysis will determine which of the three landing pages encourages user interaction, as measured by the number of 'clicks', 'viewings' and 'sessions'. Dashboard usage data will be collected through Google analytics.

DISCUSSION

This study will provide evidence of behavioural engagement and its metrics during the implementation of the CARA dashboards to optimise and sustain interaction.

TRIAL REGISTRATION

ISRCTN32783644 (Registration date: 02/01/2024).

Dashboards to Support Implementation of the Quebec Alzheimer Plan: Evaluation Study With Regional and Professional Considerations

BACKGROUND

Health organizations face the critical task of executing and overseeing comprehensive health care. To address the challenges associated with this task, evidence-based dashboards have emerged as valuable tools. Since 2016, the regional health organizations of Quebec, Canada, have been responsible for ensuring implementation of the Quebec Alzheimer Plan (QAP), a provincial plan that aims to reinforce the capacity of primary care services to detect, diagnose, and treat persons with dementia. Despite the provincial scope of the QAP, the diverse material and human resources across regions introduce variability in the interest, utility, and specific needs associated with these dashboards.

OBJECTIVE

The aim of this study was to assess the interest and utility of dashboards to support the QAP implementation, as well as to determine the needs for improving these aspects according to the perspectives of various types of professionals involved across regions.

METHODS

An evaluative study using qualitative methods was conducted within a collaborative research approach involving different stakeholders, including the ministerial advisor and the four project managers responsible for supporting the implementation of the QAP, as well as researchers/scientific advisors. To support these organizations, we developed tailored, 2-page paper dashboards, detailing quantitative data on the prevalence of dementia, the use of health services by persons with dementia, and achievements and challenges of the QAP implementation in each organization's jurisdiction. We then conducted 23 focus groups with the managers and leading clinicians involved in the implementation of the QAP of each regional health organization. Real-time notes were taken using a structured observation grid. Content analysis was conducted according to different regions (organizations with university mandates or nearby organizations, labeled "university/peripheral"; organizations for which only part of the territory is in rural areas, labeled "mixed"; and organizations in remote or isolated areas, labeled "remote/isolated") and according to different types of participants (managers, leading clinicians, and other participants).

RESULTS

Participants from organizations in all regions expressed interest in these dashboards and found them useful in several ways. However, they highlighted the need for indicators on orphan patients and other health care providers. Differences between regions were observed, particularly in the interest in continuity of care in university/peripheral regions and the need for diagnostic tools adapted to the culture in remote/isolated regions.

CONCLUSIONS

These dashboards support the implementation of an Alzheimer Plan and contribute to the emergence of a learning health care system culture. This project allows each region to increase its monitoring capacity for the implementation of the QAP and facilitates reflection among individuals locally carrying out the implementation. The perspectives expressed will guide the preparation of the next iteration of the dashboards.

Design and Evaluation of an Intensive Care Unit Dashboard Built in Response to the COVID-19 Pandemic: Semistructured Interview Study

BACKGROUND

Dashboards and interactive displays are becoming increasingly prevalent in most health care settings and have the potential to streamline access to information, consolidate disparate data sources and deliver new insights. Our research focuses on intensive care units (ICUs) which are heavily instrumented, critical care environments that generate vast amounts of data and frequently require individualized support for each patient. Consequently, clinicians experience a high cognitive load, which can translate to suboptimal performance. The global COVID-19 pandemic exacerbated this problem by generating a large number of additional hospitalizations, which necessitated a new tool that would help manage ICUs' census. In a previous study, we interviewed clinicians at the University Hospitals Bristol and Weston National Health Service Foundation Trust to capture the requirements for bespoke dashboards that would alleviate this problem.

OBJECTIVE

This study aims to design, implement, and evaluate an ICU dashboard to allow for monitoring of the high volume of patients in need of critical care, particularly tailored to high-demand situations, such as those seen during the COVID-19 pandemic.

METHODS

Building upon the previously gathered requirements, we developed a dashboard, integrated it within the ICU of a National Health Service trust, and allowed all staff to access our tool. For evaluation purposes, participants were recruited and interviewed following a 25-day period during which they were able to use the dashboard clinically. The semistructured interviews followed a topic guide aimed at capturing the usability of the dashboard, supplemented with additional questions asked post hoc to probe themes established during the interview. Interview transcripts were analyzed using a thematic analysis framework that combined inductive and deductive approaches and integrated the Technology Acceptance Model.

RESULTS

A total of 10 participants with 4 different roles in the ICU (6 consultants, 2 junior doctors, 1 nurse, and 1 advanced clinical practitioner) participated in the interviews. Our analysis generated 4 key topics that prevailed across the data: our dashboard met the usability requirements of the participants and was found useful and intuitive; participants perceived that it impacted their delivery of patient care by improving the access to the information and better equipping them to do their job; the tool was used in a variety of ways and for different reasons and tasks; and there were barriers to integration of our dashboard into practice, including familiarity with existing systems, which stifled the adoption of our tool.

CONCLUSIONS

Our findings show that the perceived utility of the dashboard had a positive impact on the clinicians' workflows in the ICU. Improving access to information translated into more efficient patient care and transformed some of the existing processes. The introduction of our tool was met with positive reception, but its integration during the COVID-19 pandemic limited its adoption into practice.

Anaesthesia and patient safety in the socio-technical operating theatre: a narrative review spanning a century

We review the development of technology in anaesthesia over the course of the past century, from the invention of the Boyle apparatus to the modern anaesthetic workstation with artificial intelligence assistance. We define the operating theatre as a socio-technical system, being necessarily comprised of human and technological parts, the ongoing development of which has led to a reduction in mortality during anaesthesia by an order of four magnitudes over a century. The remarkable technological advances in anaesthesia have been accompanied by important paradigm shifts in the approach to patient safety, and we describe the inter-relationship between technology and the human work environment in the development of such paradigm shifts, including the systems approach and organisational resilience. A better understanding of emerging technological advances and their effects on patient safety will allow anaesthesia to continue to be a leader in both patient safety and in the design of equipment and workspaces.

The Effectiveness of Interactive Dashboards to Optimise Antibiotic Prescribing in Primary Care: A Systematic Review

Governments and healthcare organisations collect data on antibiotic prescribing (AP) for surveillance. This data can support tools for visualisations and feedback to GPs using dashboards that may prompt a change in prescribing behaviour. The objective of this systematic review was to assess the effectiveness of interactive dashboards to optimise AP in primary care. Six electronic databases were searched for relevant studies up to August 2022. A narrative synthesis of findings was conducted to evaluate the intervention processes and results. Two independent reviewers assessed the relevance, risk of bias and quality of the evidence. A total of ten studies were included (eight RCTs and two non-RCTs). Overall, seven studies showed a slight reduction in AP. However, this reduction in AP when offering a dashboard may not in itself result in reductions but only when combined with educational components, public commitment or behavioural strategies. Only one study recorded dashboard engagement and showed a difference of 10% (95% CI 5% to 15%) between intervention and control. None of the studies reported on the development, pilot or implementation of dashboards or the involvement of stakeholders in design and testing. Interactive dashboards may reduce AP in primary care but most likely only when combined with other educational or behavioural intervention strategies.

Interactive process mining of cancer treatment sequences with melanoma real-world data

The growing availability of clinical real-world data (RWD) represents a formidable opportunity to complement evidence from randomized clinical trials and observe how oncological treatments perform in real-life conditions. In particular, RWD can provide insights on questions for which no clinical trials exist, such as comparing outcomes from different sequences of treatments. To this end, process mining is a particularly suitable methodology for analyzing different treatment paths and their associated outcomes. Here, we describe an implementation of process mining algorithms directly within our hospital information system with an interactive application that allows oncologists to compare sequences of treatments in terms of overall survival, progression-free survival and best overall response. As an application example, we first performed a RWD descriptive analysis of 303 patients with advanced melanoma and reproduced findings observed in two notorious clinical trials: CheckMate-067 and DREAMseq. Then, we explored the outcomes of an immune-checkpoint inhibitor rechallenge after a first progression on immunotherapy versus switching to a BRAF targeted treatment. By using interactive process-oriented RWD analysis, we observed that patients still derive long-term survival benefits from immune-checkpoint inhibitors rechallenge, which could have direct implications on treatment guidelines for patients able to carry on immune-checkpoint therapy, if confirmed by external RWD and randomized clinical trials. Overall, our results highlight how an interactive implementation of process mining can lead to clinically relevant insights from RWD with a framework that can be ported to other centers or networks of centers.