A framework for understanding, designing, developing and evaluating learning health systems

A human centered design approach to define and measure documentation quality using an EHR virtual simulation

Electronic health record (EHR) documentation serves multiple functions, including recording patient health status, enabling interprofessional communication, supporting billing, and providing data to support the quality infrastructure of a Learning Healthcare System. There is no definition and standardized method to assess documentation quality in EHRs. Using a human-centered design (HCD) approach, we define and describe a method to measure documentation quality. Documentation quality was defined as timely, accurate, user-centered, and efficient. Measurement of quality used a virtual simulated standardized patient visit via an EHR vendor platform. By observing and recording documentation efforts, nurse practitioners (NPs) (N = 12) documented the delivery of an Age-Friendly Health System (AFHS) 4Ms (what Matters, Medication, Mentation, and Mobility) clinic visit using a standardized case. Results for timely documentation indicated considerable variability in completion times of documenting the 4Ms. Accuracy varied, as there were many types of episodes of erroneous documentation and extra time in seconds in documenting the 4Ms. The type and frequency of erroneous documentation efforts were related to navigation burden when navigating to different documentation tabs. The evaluated system demonstrated poor usability, with most participants scoring between 60 and 70 on the System Usability Scale (SUS). Efficiency, measured as click burden (the number of clicks used to navigate through a software system), revealed significant variability in the number of clicks required, with the NPs averaging approximately 13 clicks above the minimum requirement. The HCD methodology used in this study to assess the documentation quality proved feasible and provided valuable information on the quality of documentation. By assessing the quality of documentation, the gathered data can be leveraged to enhance documentation, optimize user experience, and elevate the quality of data within a Learning Healthcare System.

Learning Health System to rapidly improve the implementation of a school physical activity policy

BACKGROUND

Learning Health Systems (LHS) - characterised by cycles of evidence generation and application - are increasingly recognised for their potential to improve public health interventions and optimise health impacts; however there is little evidence of their application in the context of public health practice. Here, we describe how an Australian public health unit applied a LHS approach to successfully improve a model of support for implementation of a school-based physical activity policy.

METHODS

This body of work was undertaken in the context of a strong research-practice partnership. Core LHS capabilities included: i) partnerships and stakeholder engagement; ii) workforce development and learning health communities; iii) multi-disciplinary scientific expertise; iv) practice data collection and management system; v) evidence surveillance and synthesis; and vi) governance and organisational processes of decision making. Three cycles of data generation and application were used. Within each cycle, randomised controlled trials conducted in NSW primary schools were used to generate data on the support model's effectiveness for improving schools' implementation of a government physical activity policy, its delivery costs, and process measures such as adoption and acceptability. Each type of data were analysed independently, synthesised, and then presented to a multi-disciplinary team of researchers and practitioners, in consult with stakeholders, leading to collaborative decisions for incremental improvements to the support model.

RESULTS

Cycle 1 tested the first version of the support model (composed of five implementation strategies targeting identified barriers of policy implementation) and showed the model's feasibility and efficacy for improving schools' policy implementation. Data-informed changes were made to enhance impact, including the addition of three implementation strategies to address outstanding barriers. Cycle 2 (now, testing a package of eight implementation strategies) established the model's effectiveness and cost-effectiveness for improving school's policy implementation. Data-informed changes were made to reduce delivery costs, specifically adapting the costliest strategies to reduce in-person contact from external support personnel. Cycle 3 showed that the adaptations minimised the relative cost of delivery without adversely impacting on the effect.

CONCLUSIONS

Through this process, we identified an effective, cost-effective, acceptable and scalable policy implementation support model for service delivery. This provides important information to inform or support LHS approaches for other agencies seeking to optimise the health impact of evidence-based interventions.

A protocol for the formative evaluation of the implementation of patient-reported outcome measures in child and adolescent mental health services as part of a learning health system

BACKGROUND

Mental health conditions affect one in seven young people and research suggests that current mental health services are not meeting the needs of most children and youth. Learning health systems are an approach to enhancing services through rapid, routinized cycles of continuous learning and improvement. Patient-reported outcome measures provide a key data source for learning health systems. They have also been shown to improve outcomes for patients when integrated into routine clinical care. However, implementing these measures into health systems is a challenging process. This paper describes a protocol for a formative evaluation of the implementation of patient-reported measures in a newly operational child and adolescent mental health centre in Calgary, Canada. The purpose is to optimize the collection and use of patient-reported outcome measures. Our specific objectives are to assess the implementation progress, identify barriers and facilitators to implementation, and explore patient, caregivers and clinician experiences of using these measures in routine clinical care.

METHODS

This study is a mixed-methods, formative evaluation using the Consolidated Framework for Implementation Research. Participants include patients and caregivers who have used the centre's services, as well as leadership, clinical and support staff at the centre. Focus groups and semi-structured interviews will be conducted to assess barriers and facilitators to the implementation and sustainability of the use of patient-reported outcome measures, as well as individuals' experiences with using these measures within clinical care. The data generated by the patient-reported measures over the first five months of the centre's operation will be analyzed to understand implementation progress, as well as validity of the chosen measures for the centres' population.

DISCUSSION

The findings of this evaluation will help to identify and address the factors that are affecting the successful implementation of patient-reported measures at the centre. They will inform the co-design of strategies to improve implementation with key stakeholders, which include patients, clinical staff, and leadership at the centre. To our knowledge, this is the first study of the implementation of patient-reported outcome measures in child and adolescent mental health services and our findings can be used to enhance future implementation efforts in similar settings.

Development and validity testing of a matrix to evaluate maturity of clinical pathways: a case study in Saskatchewan, Canada

BACKGROUND

Healthcare systems are transforming into learning health systems that use data-driven and research-informed approaches to achieve continuous improvement. One of these approaches is the use of clinical pathways, which are tools to standardize care for a specific population and improve healthcare quality. Evaluating the maturity of clinical pathways is necessary to inform pathway development teams and health system decision makers about required pathway revisions or implementation supports. In an effort to improve the development, implementation, and sustainability of provincial clinical pathways, we developed a clinical pathways maturity evaluation matrix. To explore the initial content and face validity of the matrix, we used it to evaluate a case pathway within a provincial health authority in Saskatchewan, Canada.

METHODS

By using iterative consensus-based processes, we gathered feedback from stakeholders including patient and family partners, policy makers, clinicians, and quality improvement specialists, to rank, retain, or remove enablers and sub-enablers of the draft matrix. We tested the matrix on the Chronic Pain Pathway (CPP) for primary care in a local pilot area and revised the matrix based on feedback from the CPP development team leader.

RESULTS

The final matrix contains five enablers (i.e., Design, Ownership and Performer, Infrastructure, Performance Management, and Culture), 20 sub-enablers, and three trajectory definitions for each sub-enabler. Supplemental documents were created for six sub-enablers. The CPP scored 15 out of 40 possible points of maturity. Although the pathway scored highest in the Design enabler (10/12), it requires more attention in several areas, specifically the Ownership and Performer and the Performance Management enablers, each of which scored zero. Additionally, the Infrastructure and Culture enablers scored 2/4 and 3/8 points, respectively. These areas of the CPP are in need of improvement in order to enhance the overall maturity of the CPP.

CONCLUSIONS

We developed a clinical pathways maturity matrix to evaluate the various dimensions of clinical pathways' development and implementation. The goals of this initial work were to develop and validate a tool to assess the maturity and readiness of new or existing pathways and to track pathways' revisions and improvements.

Dashboards to Support Implementation of the Quebec Alzheimer Plan: Evaluation Study With Regional and Professional Considerations

BACKGROUND

Health organizations face the critical task of executing and overseeing comprehensive health care. To address the challenges associated with this task, evidence-based dashboards have emerged as valuable tools. Since 2016, the regional health organizations of Quebec, Canada, have been responsible for ensuring implementation of the Quebec Alzheimer Plan (QAP), a provincial plan that aims to reinforce the capacity of primary care services to detect, diagnose, and treat persons with dementia. Despite the provincial scope of the QAP, the diverse material and human resources across regions introduce variability in the interest, utility, and specific needs associated with these dashboards.

OBJECTIVE

The aim of this study was to assess the interest and utility of dashboards to support the QAP implementation, as well as to determine the needs for improving these aspects according to the perspectives of various types of professionals involved across regions.

METHODS

An evaluative study using qualitative methods was conducted within a collaborative research approach involving different stakeholders, including the ministerial advisor and the four project managers responsible for supporting the implementation of the QAP, as well as researchers/scientific advisors. To support these organizations, we developed tailored, 2-page paper dashboards, detailing quantitative data on the prevalence of dementia, the use of health services by persons with dementia, and achievements and challenges of the QAP implementation in each organization's jurisdiction. We then conducted 23 focus groups with the managers and leading clinicians involved in the implementation of the QAP of each regional health organization. Real-time notes were taken using a structured observation grid. Content analysis was conducted according to different regions (organizations with university mandates or nearby organizations, labeled "university/peripheral"; organizations for which only part of the territory is in rural areas, labeled "mixed"; and organizations in remote or isolated areas, labeled "remote/isolated") and according to different types of participants (managers, leading clinicians, and other participants).

RESULTS

Participants from organizations in all regions expressed interest in these dashboards and found them useful in several ways. However, they highlighted the need for indicators on orphan patients and other health care providers. Differences between regions were observed, particularly in the interest in continuity of care in university/peripheral regions and the need for diagnostic tools adapted to the culture in remote/isolated regions.

CONCLUSIONS

These dashboards support the implementation of an Alzheimer Plan and contribute to the emergence of a learning health care system culture. This project allows each region to increase its monitoring capacity for the implementation of the QAP and facilitates reflection among individuals locally carrying out the implementation. The perspectives expressed will guide the preparation of the next iteration of the dashboards.

Deriving Treatment Decision Support From Dutch Electronic Health Records by Exploring the Applicability of a Precision Cohort-Based Procedure for Patients With Type 2 Diabetes Mellitus: Precision Cohort Study

BACKGROUND

The rapidly increasing availability of medical data in electronic health records (EHRs) may contribute to the concept of learning health systems, allowing for better personalized care. Type 2 diabetes mellitus was chosen as the use case in this study.

OBJECTIVE

This study aims to explore the applicability of a recently developed patient similarity-based analytics approach based on EHRs as a candidate data analytical decision support tool.

METHODS

A previously published precision cohort analytics workflow was adapted for the Dutch primary care setting using EHR data from the Nivel Primary Care Database. The workflow consisted of extracting patient data from the Nivel Primary Care Database to retrospectively generate decision points for treatment change, training a similarity model, generating a precision cohort of the most similar patients, and analyzing treatment options. This analysis showed the treatment options that led to a better outcome for the precision cohort in terms of clinical readouts for glycemic control.

RESULTS

Data from 11,490 registered patients diagnosed with type 2 diabetes mellitus were extracted from the database. Treatment-specific filter cohorts of patient groups were generated, and the effect of past treatment choices in these cohorts was assessed separately for glycated hemoglobin and fasting glucose as clinical outcome variables. Precision cohorts were generated for several individual patients from the filter cohorts. Treatment options and outcome analyses were technically well feasible but in general had a lack of statistical power to demonstrate statistical significance for treatment options with better outcomes.

CONCLUSIONS

The precision cohort analytics workflow was successfully adapted for the Dutch primary care setting, proving its potential for use as a learning health system component. Although the approach proved technically well feasible, data size limitations need to be overcome before application for clinical decision support becomes realistically possible.

Promoting knowledge translation: An ecosystem approach to evidence in health

The dissemination and implementation of evidence in health contexts have been a concern of several international organizations responsible for recommending actions to health policymakers. World Health Organization has been advocating for an ecosystem of evidence to improve clinical practice and health professional education. Thus, in this article, we address the challenges to developing the evidence ecosystem from the point of view of health professional education, considering the contexts of practice and teaching, focused on knowledge translation. There are three pivotal challenges: producing qualified knowledge; adequate communication of the synthesized evidence; and institutional policy to sustain the implemented evidence in continuous and updated flow. The evidence ecosystem helps to understand these flows between the production and implementation of knowledge, based on the capacity and resources of different health systems. It needs to be developed in the field of health professional education, feedback in the contexts of practice and teaching, to contribute to third-generation knowledge being used by different users of health services.

Building Learning Healthcare Systems for Critical Care Medicine

Learning healthcare systems are an evolving way of integrating informatics, analytics, and continuous improvement into daily practice in healthcare. This article discusses strategies to build learning healthcare systems for critical care medicine.

Building a NICU quality & safety infrastructure

The American Academy of Pediatrics (AAP) Standards for Levels of Neonatal Care, published in 2023, highlights key components of a Neonatal Patient Safety and Quality Improvement Program (NPSQIP). A comprehensive Neonatal Intensive Care Unit (NICU) quality and safety infrastructure (QSI) is based on four foundational domains: quality improvement, quality assurance, safety culture, and clinical guidelines. This paper serves as an operational guide for NICU clinical leaders and quality champions to navigate these domains and develop their local QSI to include the AAP NPSQIP standards.

Progress with the Learning Health System 2.0: a rapid review of Learning Health Systems' responses to pandemics and climate change

BACKGROUND

Pandemics and climate change each challenge health systems through increasing numbers and new types of patients. To adapt to these challenges, leading health systems have embraced a Learning Health System (LHS) approach, aiming to increase the efficiency with which data is translated into actionable knowledge. This rapid review sought to determine how these health systems have used LHS frameworks to both address the challenges posed by the COVID-19 pandemic and climate change, and to prepare for future disturbances, and thus transition towards the LHS2.0.

METHODS

Three databases (Embase, Scopus, and PubMed) were searched for peer-reviewed literature published in English in the five years to March 2023. Publications were included if they described a real-world LHS's response to one or more of the following: the COVID-19 pandemic, future pandemics, current climate events, future climate change events. Data were extracted and thematically analyzed using the five dimensions of the Institute of Medicine/Zurynski-Braithwaite's LHS framework: Science and Informatics, Patient-Clinician Partnerships, Continuous Learning Culture, Incentives, and Structure and Governance.

RESULTS

The search yielded 182 unique publications, four of which reported on LHSs and climate change. Backward citation tracking yielded 13 additional pandemic-related publications. None of the climate change-related papers met the inclusion criteria. Thirty-two publications were included after full-text review. Most were case studies (n = 12, 38%), narrative descriptions (n = 9, 28%) or empirical studies (n = 9, 28%). Science and Informatics (n = 31, 97%), Continuous Learning Culture (n = 26, 81%), Structure and Governance (n = 23, 72%) were the most frequently discussed LHS dimensions. Incentives (n = 21, 66%) and Patient-Clinician Partnerships (n = 18, 56%) received less attention. Twenty-nine papers (91%) discussed benefits or opportunities created by pandemics to furthering the development of an LHS, compared to 22 papers (69%) that discussed challenges.

CONCLUSIONS

An LHS 2.0 approach appears well-suited to responding to the rapidly changing and uncertain conditions of a pandemic, and, by extension, to preparing health systems for the effects of climate change. LHSs that embrace a continuous learning culture can inform patient care, public policy, and public messaging, and those that wisely use IT systems for decision-making can more readily enact surveillance systems for future pandemics and climate change-related events.

TRIAL REGISTRATION

PROSPERO pre-registration: CRD42023408896.

Prioritizing Patient Reported Outcome Measures (PROMs) to use in the clinical care of youth living with mental health concerns: a nominal group technique study

BACKGROUND

In the past few decades, particularly in the mental health setting, there has been growing interest in using Patient Reported Outcome Measures (PROMs) to assess the efficacy of the treatments in healthcare systems. Despite recent initiatives for global harmonization, there remains a lack of consensus on which PROMs are best practice and appropriate. Engagement of the service users, such as patients and family members/caregivers, is vital at this stage to ensure the selected PROMs are feasible, relevant, and acceptable to them. This study aimed to prioritize PROMs by youth and family/caregiver based on feasibility, relevance, and overall importance to be used in the clinical care of youth living with anxiety and/or depression.

METHODS

Ten validated and widely used PROMs were presented to the patients and family/caregivers. Nominal group techniques were employed to prioritize the PROMs based on feasibility, relevance, and overall importance.

RESULTS

For patients and families/caregivers, the PROMs, Revised Child Anxiety and Depression Scale (RCAD 25), and The Young Person's Core (YP-CORE) were the highest priorities. Both felt that RCAD 25 was comprehensive, short, easy, and quick to complete, whereas regarding YP-CORE, patients and family/caregivers thought it was also short and relevant. Due to some specific concerns, the Strength and Difficulties Questionnaire and Child Health Questionnaire were the lowest prioritized by patients and family/caregivers.

CONCLUSION

It is of utmost importance that patient's and family/caregivers' voices or opinions are considered while selecting and implementing PROMs in mental health settings. Our study provides practical recommendations around measures best suited to achieve this.

Harnessing the potential of data-driven strategies to optimise transfusion practice

No one doubts the significant variation in the practice of transfusion medicine. Common examples are the variability in transfusion thresholds and the use of tranexamic acid for surgery with likely high blood loss despite evidence-based standards. There is a long history of applying different strategies to address this variation, including education, clinical guidelines, audit and feedback, but the effectiveness and cost-effectiveness of these initiatives remains unclear. Advances in computerised decision support systems and the application of novel electronic capabilities offer alternative approaches to improving transfusion practice. In England, the National Institute for Health and Care Research funded a Blood and Transplant Research Unit (BTRU) programme focussing on 'A data-enabled programme of research to improve transfusion practices'. The overarching aim of the BTRU is to accelerate the development of data-driven methods to optimise the use of blood and transfusion alternatives, and to integrate them within routine practice to improve patient outcomes. One particular area of focus is implementation science to address variation in practice.

Learning health system benefits: Development and initial validation of a framework

Introduction

Implementation of research findings in clinical practice often is not realized or only partially achieved, and if so, with a significant delay. Learning health systems (LHSs) hold promise to overcome this problem by embedding clinical research and evidence-based best practices into care delivery, enabling innovation and continuous improvement. Implementing an LHS is a complex process that requires participation and resources of a wide range of stakeholders, including healthcare leaders, clinical providers, patients and families, payers, and researchers. Engaging these stakeholders requires communicating clear, tangible value propositions. Existing models identify broad categories of benefits but do not explicate the full range of benefits or ways they can manifest in different organizations.

Methods

To develop such a framework, a working group with representatives from six Clinical and Translational Science Award (CTSA) hubs reviewed existing literature on LHS characteristics, models, and goals; solicited expert input; and applied the framework to their local LHS experiences.

Results

The Framework of LHS Benefits includes six categories of benefits (quality, safety, equity, patient satisfaction, reputation, and value) relevant for a range of stakeholders and defines key concepts within each benefit. Applying the framework to five LHS case examples indicated preliminary face validity across varied LHS approaches and revealed three dimensions in which the framework is relevant: defining goals of individual LHS projects, facilitating collaboration based on shared values, and establishing guiding tenets of an LHS program or mission.

Conclusion

The framework can be used to communicate the value of an LHS to different stakeholders across varied contexts and purposes, and to identify future organizational priorities. Further validation will contribute to the framework's evolution and support its potential to inform the development of tools to evaluate LHS impact.

21st century (clinical) decision support in nursing and allied healthcare. Developing a learning health system: a reasoned design of a theoretical framework

In this paper, we present a framework for developing a Learning Health System (LHS) to provide means to a computerized clinical decision support system for allied healthcare and/or nursing professionals. LHSs are well suited to transform healthcare systems in a mission-oriented approach, and is being adopted by an increasing number of countries. Our theoretical framework provides a blueprint for organizing such a transformation with help of evidence based state of the art methodologies and techniques to eventually optimize personalized health and healthcare. Learning via health information technologies using LHS enables users to learn both individually and collectively, and independent of their location. These developments demand healthcare innovations beyond a disease focused orientation since clinical decision making in allied healthcare and nursing is mainly based on aspects of individuals' functioning, wellbeing and (dis)abilities. Developing LHSs depends heavily on intertwined social and technological innovation, and research and development. Crucial factors may be the transformation of the Internet of Things into the Internet of FAIR data & services. However, Electronic Health Record (EHR) data is in up to 80% unstructured including free text narratives and stored in various inaccessible data warehouses. Enabling the use of data as a driver for learning is challenged by interoperability and reusability.To address technical needs, key enabling technologies are suitable to convert relevant health data into machine actionable data and to develop algorithms for computerized decision support. To enable data conversions, existing classification and terminology systems serve as definition providers for natural language processing through (un)supervised learning.To facilitate clinical reasoning and personalized healthcare using LHSs, the development of personomics and functionomics are useful in allied healthcare and nursing. Developing these omics will be determined via text and data mining. This will focus on the relationships between social, psychological, cultural, behavioral and economic determinants, and human functioning.Furthermore, multiparty collaboration is crucial to develop LHSs, and man-machine interaction studies are required to develop a functional design and prototype. During development, validation and maintenance of the LHS continuous attention for challenges like data-drift, ethical, technical and practical implementation difficulties is required.