A maturity grid assessment tool for learning networks

Tools and frameworks for evaluating the implementation of learning health systems: a scoping review

INTRODUCTION

Despite increased interest in learning health systems (LHS), a paucity of guidance and tools for evaluating LHS implementation exists. To address this, we aim to undertake a scoping review on existing tools and evaluation of exemplars of LHS implementation.

METHODS

We conducted a scoping review of peer-reviewed studies within Scopus, EMBASE, MEDLINE, and MEDLINE in-process that described (1) the evaluation of the implementation of an operating LHS or (2) the development of a framework or tool to facilitate this evaluation. Anima, basic research, abstracts, non-English language articles, and publications before 2018 were excluded. All study designs were considered.

FINDINGS

From 1300 studies initially identified, 4 were eligible, revealing three tools with nine implementation evaluation examples. The identified tools shared constructs which were evaluated, including: Stakeholders, Data, Research Evidence, Implementation, and Sociotechnical Infrastructure. However, there was divergence in evaluation methodology. Tools ranged from a five-point numerical rating system for process maturity with a radar chart called the Network Maturity Grid (NMG); the Kaiser Permanente Washington (KPWA) LHS Logic Model, which provides a broad list of constructs and sample measures relevant to LHS operations; and finally LADDERS, a simple tool or form-based template designed for consistent evaluation over time. The NMG tool was the most mature in terms of adaptation and adoption. Notably, two (NMG and the KPWA LHS Logic Model) out of three tools conceptualized the LHS as a suite of processes and devised tools were processes that linked these constructs.

IMPLICATIONS FOR TOOLKIT DEVELOPMENT

The evaluation of LHS implementation remains an under explored area of investigation, as this scoping review found only three tools for LHS implementation evaluation. Our findings indicate a need for further empirical research in this area and suggest early consensus in constructs that need to be considered during evaluation.

Funding Learning Health System Research: Challenges and Strategies

PURPOSE

A growing number of health systems are establishing learning health system (LHS) programs, where research focuses on rapidly improving the health system's internal operations and performance. The authors examine funding challenges facing such initiatives and identify strategies for managing tensions between reliance on external research funding and directly contributing to improvement and learning within the researchers' own system.

METHOD

Qualitative case studies of LHS research programs in 5 health systems were performed via 38 semistructured interviews (October 2019-April 2021) with 35 diverse respondents. Inductive and deductive rapid qualitative analysis supported interview, system-level, and cross-system summaries and analysis.

RESULTS

External funding awards to LHS researchers facilitated some internal improvement and learning, scientific advancements, and the reputation of researchers and their systems, but reliance on external funding also challenged researchers' responsiveness to concerns of system leaders, managers, practitioners, and system needs. Gaps between external funding requirements and internally focused projects arose in objectives, practical applicability, audiences, timetables, routines, skill sets, and researchers' careers. To contribute more directly to system improvement, LHS researchers needed to collaborate with clinicians and other nonresearchers and pivot between long research studies and shorter, dynamic improvement, evaluation, and data analysis projects. With support from system executives, LHS program leaders employed several strategies to enhance researchers' internal contributions. They aligned funded-research topics with long-term system needs, obtained internal funding for implementing and sustaining practice change, and diversified funding sources.

CONCLUSIONS

To foster LHS research contributions to internal system learning and improvement, LHS program leaders need to manage tensions between concentrating on externally funded research and fulfilling their mission of providing research-based services to their own system. Health system executives can support LHS programs by setting clear goals for them; appropriately staffing, budgeting, and incentivizing LHS researchers; and developing supportive, system-wide teamwork, skill development programs, and data infrastructures.

The Cystic Fibrosis Learning Network: A mixed methods evaluation of program goals, attributes, and impact

Introduction

The Cystic Fibrosis (CF) Foundation sponsored the design, pilot testing, and implementation of the CF Learning Network (CFLN) to explore how the Foundation's Care Center Network (CCN) could become a learning health system. Six years after the design, the Foundation commissioned a formative mixed methods evaluation of the CFLN to assess: CFLN participants' understanding of program goals, attributes, and perceptions of current and future impact.

Methods

We performed semi-structured interviews with CFLN participants to identify perceived goals, attributes, and impact of the network. Following thematic analyses, we developed and distributed a survey to CFLN members and a matched sample of CCN programs to understand whether the themes were unique to the CFLN.

Results

Interviews with 24 CFLN participants were conducted. Interviewees identified the primary CFLN goal as improving outcomes for people living with CF, with secondary goals of providing training in quality improvement (QI), creating a learning community, engaging all stakeholders in improvement, and spreading best practices to the CCN. Project management, use of data, common QI methods, and the learning community were seen as critical to success. Survey responses were collected from 103 CFLN members and 25 CCN members. The data revealed that CFLN respondents were more likely than CCN respondents to connect with other CF programs, routinely use data for QI, and engage patient and family partners in QI.

Conclusions

Our study suggests that the CFLN provides value beyond that achieved by the CCN. Key questions remain about whether spread of the CFLN could improve outcomes for more people living with CF.

Establishing a Cystic Fibrosis Learning Network: Interventions to promote collaboration and data-driven improvement at scale

Introduction

A learning health network is a type of learning health system in which stakeholders use network organization to improve health and health care. Building on existing resources in the cystic fibrosis (CF) community, the Cystic Fibrosis Learning Network (CFLN) was designed to improve medical outcomes and quality of life through an intentional focus on achieving reliable evidence-based chronic care delivery and creating a system for data-driven collaborative learning.

Methods

We describe the development and growth of the CFLN considering six domains of a Network Maturity Grid: system leadership; governance and policy management; quality improvement (QI); engagement and community building; data and analytics; and research. We illustrate the impact of the CFLN experience on chronic care processes and indicators of collaborative infrastructure.

Results

The CFLN represents 36 accredited care centers in the CF Foundation Care Center Network caring for over 6300 patients. Of 6779 patient clinical care visits/quarter, 77% are entered into the CF Foundation Patient Registry within 30 days, providing timely means to track outcomes. Collaborative visit planning is occurring in 93% of clinical care visits to share agenda setting with patients and families. Almost all CFLN teams (94%, n = 34) have a patient/family partner (PFP), and 74% of PFPs indicate they are actively participating, taking ownership of, or leading QI initiatives with the interdisciplinary care team. In 2022, 97% of centers reported completing 1-13 improvement cycles per month, and 82% contributed to monthly QI progress reports to share learning.

Conclusion

The CFLN is a maturing, collaborative infrastructure. CFLN centers practice at an advanced level of coproduction. The CFLN fosters interdisciplinary and PFP leadership and the performance of consistent data-driven improvement cycles. CFLN centers are positioned to respond to rapid changes in evidence-based care and advance the practice of QI and implementation science on a broader scale.

A framework for understanding, designing, developing and evaluating learning health systems

Introduction

A Learning Health System is not a technical project. It is the evolution of an existing health system into one capable of learning from every patient. This paper outlines a recently published framework intended to aid the understanding, design, development and evaluation of Learning Health Systems.

Methods

This work extended an existing repository of Learning Health System evidence, adding five more workshops. The total was subjected to thematic analysis, yielding a framework of elements important to understanding, designing, developing and evaluating Learning Health Systems. Purposeful literature reviews were conducted on each element. The findings were revised following a review by a group of international experts.

Results

The resulting framework was arranged around four questions:What is our rationale for developing a Learning Health System?There can be many reasons for developing a Learning Health System. Understanding these will guide its development.What sources of complexity exist at the system and the intervention level?An understanding of complexity is central to making Learning Health Systems work. The non-adoption, abandonment, scale-up, spread and sustainability framework was utilised to help understand and manage it.What strategic approaches to change do we need to consider?A range of strategic issues must be addressed to enable successful change in a Learning Health System. These include, strategy, organisational structure, culture, workforce, implementation science, behaviour change, co-design and evaluation.What technical building blocks will we need?A Learning Health System must capture data from practice, turn it into knowledge and apply it back into practice. There are many methods to achieve this and a range of platforms to help.

Discussion

The results form a framework for understanding, designing, developing and evaluating Learning Health Systems at any scale.

Conclusion

It is hoped that this framework will evolve with use and feedback.

How Dissemination and Implementation Science Can Contribute to the Advancement of Learning Health Systems

Many health systems are working to become learning health systems (LHSs), which aim to improve the value of health care by rapidly, continuously generating evidence to apply to practice. However, challenges remain to advance toward the aspirational goal of becoming a fully mature LHS. While some important challenges have been well described (i.e., building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data), other key challenges are underrecognized, including balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on external validity and reproducibility (generalizability), and designing for sustainability. Many LHSs focus on continuous learning cycles, but with limited consideration of issues related to the rapidity of these learning cycles, as well as the sustainability or generalizability of solutions. Some types of data have been consistently underrepresented, including patient-reported outcomes and preferences, social determinants, and behavioral and environmental data, the absence of which can exacerbate health disparities. A promising approach to addressing many challenges that LHSs face may be found in dissemination and implementation (D&I) science. With an emphasis on multilevel dynamic contextual factors, representation of implementation partner engagement, pragmatic research, sustainability, and generalizability, D&I science methods can assist in overcoming many of the challenges facing LHSs. In this article, the authors describe the current state of LHSs and challenges to becoming a mature LHS, propose solutions to current challenges, focusing on the contributions of D&I science with other methods, and propose key components and characteristics of a mature LHS model that others can use to plan and develop their LHSs.

Training the next generation of learning health system scientists

Introduction

The learning health system (LHS) aligns science, informatics, incentives, stakeholders, and culture for continuous improvement and innovation. The Agency for Healthcare Research and Quality and the Patient‐Centered Outcomes Research Institute designed a K12 initiative to grow the number of LHS scientists. We describe approaches developed by 11 funded centers of excellence (COEs) to promote partnerships between scholars and health system leaders and to provide mentored research training.

Methods

Since 2018, the COEs have enlisted faculty, secured institutional resources, partnered with health systems, developed and implemented curricula, recruited scholars, and provided mentored training. Program directors for each COE provided descriptive data on program context, scholar characteristics, stakeholder engagement, scholar experiences with health system partnerships, roles following program completion, and key training challenges.

Results

To date, the 11 COEs have partnered with health systems to train 110 scholars. Nine (82%) programs partner with a Veterans Affairs health system and 9 (82%) partner with safety net providers. Clinically trained scholars (n = 87; 79%) include 70 physicians and 17 scholars in other clinical disciplines. Non‐clinicians (n = 29; 26%) represent diverse fields, dominated by population health sciences. Stakeholder engagement helps scholars understand health system and patient/family needs and priorities, enabling opportunities to conduct embedded research, improve outcomes, and grow skills in translating research methods and findings into practice. Challenges include supporting scholars through roadblocks that threaten to derail projects during their limited program time, ranging from delays in access to data to COVID‐19‐related impediments and shifts in organizational priorities.

Conclusions

Four years into this novel training program, there is evidence of scholars' accomplishments, both in traditional academic terms and in terms of moving along career trajectories that hold the potential to lead and accelerate transformational health system change. Future LHS training efforts should focus on sustainability, including organizational support for scholar activities.

The SHOnet learning health system: Infrastructure for continuous learning in pediatric rehabilitation

INTRODUCTION

To describe the development and implementation of learning health system (LHS) infrastructure for a pediatric specialty care health system to support LHS research in pediatric rehabilitation settings.

METHODS

An existing pediatric common data model (eg, PEDSnet) of standardized medical terminologies for research was expanded and leveraged for this stud, and applied to SHOnet, a clinical research data resource consisting of deidentified data extracted from the electronic health record (EHR) from the Shriners Hospitals for Children speacialty pediatric health care system. We mapped EHR data for laboratory, procedures, drugs, and conditions to standardized vocabularies including ICD-10, CPT, RxNorm, and LOINC to the common data model using an established extraction-transformation-loading process. Rigorous quality checks were conducted to ensure a high degree of data conformance, completeness, and plausibility. SHOnet data elements from all sources are de-identified and the server is managed by the SHC Information Systems Department. SHOnet data are refreshed monthly and data elements are continually expanded based on new research endeavors.

INTERVENTIONS

Not applicable.

RESULTS

The Shriners Health Outcomes Network (SHOnet) includes data for over 10 000 distinct observational data elements based on over two million patient encounters between 2011 and present.

CONCLUSION

The systematic process to develop SHOnet is replicable and flexible for other pediatric rehabilitation research settings interested in building out their LHS capabilities. Challenges and facilitators may arise for building such LHS infrastructure for rehabilitation in areas of (a) data capture, curation, query, and governance, (b) generating knowledge from data, and (c) dissemination and implementation of new institutional knowledge. Further research studies are needed to evaluate these data resources for scalable system-learning endeavors.SHOnet is an exemplar of an LHS for rehabilitation and specialty care settings. The success of an LHS is dependent on engagement of multiple stakeholders, shared governance, effective knowledge translation, and deep commitment to long-term strategies for engaging clinicians, administration, and families in leveraging knowledge to improve clinical outcomes.

Toward an ontology of collaborative learning healthcare systems

Objective

To establish a basis for a domain ontology ‐ a formal, explicit specification of a shared conceptualization ‐ of collaborative learning healthcare systems (CLHSs) in order to facilitate measurement, explanation, and improvement.

Methods

We adapted the “Methontology” approach to begin building an ontology of CLHSs. We specified the purpose of an ontology, acquired domain knowledge via literature review, conceptualized a common framework of CLHSs using a grounded approach, refined these concepts based on expert panel input, and illustrated concept application via four cases.

Results

The set of concepts identified as important to include in an ontology includes goals, values, structure, actors, environment, and products. To establish this set of concepts, we gathered input from content experts in two ways. First, expert panel methods were used to elicit feedback on these concepts and to test the elicitation of terms for the vocabulary of the Values concept. Second, from these discussions we developed a mapping exercise to test the intuitiveness of the concepts, requesting that network leaders from four CLHSs complete a mapping exercise to associate characteristics of their networks with the high‐level concepts, building the vocabulary for each concept in a grounded fashion. We also solicited feedback from these participants on the experience of completing the mapping exercise, finding that the exercise is acceptable and could aid in CLHS development and collaboration. Respondents identified opportunities to improve the operational definitions of each concept to ensure that corresponding vocabularies are distinct and non‐overlapping.

Discussion

Our results provide a foundation for developing a formal, explicit shared conceptualization of CLHSs. Once developed, such a tool can be useful for measurement, explanation, and improvement. Further work, including alignment to a top‐level ontology, expanding the vocabulary, and defining relations between vocabulary is required to formally build out an ontology for these uses.

Collaborative conversations during the time of COVID-19: Building a "meta"-learning community

PROBLEM

COVID-19 created new research, clinical, educational, and personal challenges, while simultaneously separating work teams who were under work-from-home restrictions. Addressing these challenges required new forms of collaborative groups.

APPROACH

To support the department community and the rapid sharing of new research, educational, clinical, and personal efforts, a Core Team from the Department of Learning Health Sciences at the University of Michigan developed a meeting series called the COVID Conversations. This Experience Report shares the organizational structure of the COVID Conversations, proposes a comparison to traditional Learning Communities, and reports the results of a questionnaire that gathered details about department members' COVID-related activities.

OUTCOMES

We identify and describe salient similarities and differences between the COVID Conversations and the characteristics of Learning Communities. We also developed and piloted a taxonomy for characterizing LHS research projects that may be further developed for use in Learning Community planning, in conjunction with other maturity grids and ontologies. We propose the term "Meta-Learning Community" to describe the structure and function of the COVID Conversations.

NEXT STEPS

In academic medicine, remote work, telemedicine, and virtual learning may be here to stay. The COVID Conversations constitute a distinct and innovative form of collaborative work in which separate teams addressing distinct goals, yet sharing a common passion to tackle the issues brought by the pandemic, are able to share experiences and learn from one other. The challenges of COVID-19 have made evident the need for multiple forms of organizing teamwork, and our study contributes the notion of a "Meta"-Learning Community as a new form of collaborative work.

The Indiana Learning Health System Initiative: Early experience developing a collaborative, regional learning health system

INTRODUCTION

Learning health systems (LHSs) are usually created and maintained by single institutions or healthcare systems. The Indiana Learning Health System Initiative (ILHSI) is a new multi-institutional, collaborative regional LHS initiative led by the Regenstrief Institute (RI) and developed in partnership with five additional organizations: two Indiana-based health systems, two schools at Indiana University, and our state-wide health information exchange. We report our experiences and lessons learned during the initial 2-year phase of developing and implementing the ILHSI.

METHODS

The initial goals of the ILHSI were to instantiate the concept, establish partnerships, and perform LHS pilot projects to inform expansion. We established shared governance and technical capabilities, conducted a literature review-based and regional environmental scan, and convened key stakeholders to iteratively identify focus areas, and select and implement six initial joint projects.

RESULTS

The ILHSI successfully collaborated with its partner organizations to establish a foundational governance structure, set goals and strategies, and prioritize projects and training activities. We developed and deployed strategies to effectively use health system and regional HIE infrastructure and minimize information silos, a frequent challenge for multi-organizational LHSs. Successful projects were diverse and included deploying a Fast Healthcare Interoperability Standards (FHIR)-based tool across emergency departments state-wide, analyzing free-text elements of cross-hospital surveys, and developing models to provide clinical decision support based on clinical and social determinants of health. We also experienced organizational challenges, including changes in key leadership personnel and varying levels of engagement with health system partners, which impacted initial ILHSI efforts and structures. Reflecting on these early experiences, we identified lessons learned and next steps.

CONCLUSIONS

Multi-organizational LHSs can be challenging to develop but present the opportunity to leverage learning across multiple organizations and systems to benefit the general population. Attention to governance decisions, shared goal setting and monitoring, and careful selection of projects are important for early success.

A science of collaborative learning health systems

INTRODUCTION

Improving the U.S. healthcare system and health outcomes is one of the most pressing public health challenges of our time. Previously described Collaborative Learning Health Systems (CLHSs) are a promising approach to outcomes improvement. In order to fully realize this promise, a deeper understanding of this phenomenon is necessary.

METHODS

We drew on our experience over the past decade with CLHSs as well as qualitative literature review to answer three questions: What kind of phenomena are CLHSs? and what is an appropriate scientific approach? How might we frame CLHSs conceptually? What are potential mechanisms of action?

RESULTS

CLHSs are complex adaptive systems in which all stakeholders are able to collaborate, at scale, to create and share resources to satisfy a variety of needs. This is accomplished by providing infrastructure and services that enable stakeholders to act on their inherent motivations. This framing has implications for both research and practice.

CONCLUSION

Articulating this framework and potential mechanisms of action should facilitate research to test and refine hypotheses as well as guide practice to develop and optimize this promising approach to improving healthcare systems.

A collaborative learning health system agent-based model: Computational and face validity

INTRODUCTION

Improving the healthcare system is a major public health challenge. Collaborative learning health systems (CLHS) - network organizations that allow all healthcare stakeholders to collaborate at scale - are a promising response. However, we know little about CLHS mechanisms of actions, nor how to optimize CLHS performance. Agent-based models (ABM) have been used to study a variety of complex systems. We translate the conceptual underpinnings of a CLHS to a computational model and demonstrate initial computational and face validity.

METHODS

CLHSs are organized to allow stakeholders (patients and families, clinicians, researchers) to collaborate, at scale, in the production and distribution of information, knowledge, and know-how for improvement. We build up a CLHS ABM from a population of patient- and doctor-agents, assign them characteristics, and set them into interaction, resulting in engagement, information, and knowledge to facilitate optimal treatment selection. To assess computational and face validity, we vary a single parameter - the degree to which patients influence other patients - and trace its effects on patient engagement, shared knowledge, and outcomes.

RESULTS

The CLHS ABM, developed in Python and using the open-source modeling framework Mesa, is delivered as a web application. The model is simulated on a cloud server and the user interface is a web browser using Python and Plotly Dash. Holding all other parameters steady, when patient influence increases, the overall patient population activation increases, leading to an increase in shared knowledge, and higher median patient outcomes.

CONCLUSIONS

We present the first theoretically-derived computational model of CLHSs, demonstrating initial computational and face validity. These preliminary results suggest that modeling CLHSs using an ABM is feasible and potentially valid. A well-developed and validated computational model of the health system may have profound effects on understanding mechanisms of action, potential intervention targets, and ultimately translation to improved outcomes.