1
|
Kilbourne AM, Borsky AE, O'Brien RW, Braganza MZ, Garrido MM. The foundational science of learning health systems. Health Serv Res 2024. [PMID: 39165034 DOI: 10.1111/1475-6773.14374] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/22/2024] Open
Affiliation(s)
- Amy M Kilbourne
- Office of Research and Development, Veterans Health Administration, U.S. Department of Veterans Affairs, Washington, DC, USA
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan, USA
| | - Amanda E Borsky
- Office of Research and Development, Veterans Health Administration, U.S. Department of Veterans Affairs, Washington, DC, USA
| | - Robert W O'Brien
- Office of Research and Development, Veterans Health Administration, U.S. Department of Veterans Affairs, Washington, DC, USA
| | - Melissa Z Braganza
- Office of Research and Development, Veterans Health Administration, U.S. Department of Veterans Affairs, Washington, DC, USA
| | - Melissa M Garrido
- Partnered Evidence-based Policy Resource Center, VA Boston Healthcare System, Boston, Massachusetts, USA
- Department of Health Law, Policy & Management, Boston University School of Public Health, Boston, Massachusetts, USA
- Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA
| |
Collapse
|
2
|
van Vooren N, de Weger E, de Bruin J, Baan C. Improving the learning capacity of regional health systems for their transformation towards health and well-being systems: a qualitative study of ten Dutch regions. J Health Organ Manag 2024; 38:280-296. [PMID: 39308093 PMCID: PMC11346209 DOI: 10.1108/jhom-06-2023-0187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2023] [Revised: 04/09/2024] [Accepted: 04/16/2024] [Indexed: 09/26/2024]
Abstract
PURPOSE There is growing recognition that transformation of healthcare systems towards health and well-being systems requires a continuous learning process. This explorative study aims to gain insight into the experiences with and investment in these learning processes within regional partnerships for health and in what they need to enhance their learning capacity to use the learning for transformation. DESIGN/METHODOLOGY/APPROACH 17 interviews were held with programme managers, data scientists, trusted advisors and a citizen representative, all involved in the learning process on a regional level in ten Dutch regional partnerships. The interviews were inductively and thematically analysed, focusing on the experiences and perceptions underlying the learning processes. FINDINGS Regional partnerships invest in learning processes by organizing interactions between different groups of stakeholders and by reflecting on specific themes or on a region-wide level. Difficulty was found in region-wide reflection and in enhancing the learning capacity within the partnerships. Further enhancing the learning capacity required: (1) Investment in (the use of) expertise for translating learning outcomes into concrete action; (2) Leadership for change, underpinned by a shared sense of urgency to learn for transformation and (3) A facilitative environment for change which is both based on facilitative system structures and a basis of trust and commitment to learn and adapt. ORIGINALITY/VALUE The study highlighted the difficulty of learning on a region-wide level and the struggle to apply this learning for transformation. It provides insights into how learning processes and learning capacity can be further improved.
Collapse
Affiliation(s)
- Natascha van Vooren
- Center for Nutrition, Prevention and Health
Services, Rijksinstituut voor Volksgezondheid en
Milieu, Bilthoven, Netherlands
- Tranzo, Tilburg School of
Social and Behavioral Sciences, Tilburg
University, Tilburg, Netherlands
| | - Esther de Weger
- Athena Institute, Vrije
Universiteit Amsterdam, Amsterdam, Netherlands
| | - Josefien de Bruin
- Center for Nutrition, Prevention and Health
Services, Rijksinstituut voor Volksgezondheid en
Milieu, Bilthoven, Netherlands
- Tranzo, Tilburg School of
Social and Behavioral Sciences, Tilburg
University, Tilburg, Netherlands
| | - Caroline Baan
- Tranzo, Tilburg School of
Social and Behavioral Sciences, Tilburg
University, Tilburg, Netherlands
| |
Collapse
|
3
|
Rajit D, Reeder S, Johnson A, Enticott J, Teede H. Tools and frameworks for evaluating the implementation of learning health systems: a scoping review. Health Res Policy Syst 2024; 22:95. [PMID: 39107779 PMCID: PMC11302020 DOI: 10.1186/s12961-024-01179-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2024] [Accepted: 07/03/2024] [Indexed: 08/10/2024] Open
Abstract
INTRODUCTION Despite increased interest in learning health systems (LHS), a paucity of guidance and tools for evaluating LHS implementation exists. To address this, we aim to undertake a scoping review on existing tools and evaluation of exemplars of LHS implementation. METHODS We conducted a scoping review of peer-reviewed studies within Scopus, EMBASE, MEDLINE, and MEDLINE in-process that described (1) the evaluation of the implementation of an operating LHS or (2) the development of a framework or tool to facilitate this evaluation. Anima, basic research, abstracts, non-English language articles, and publications before 2018 were excluded. All study designs were considered. FINDINGS From 1300 studies initially identified, 4 were eligible, revealing three tools with nine implementation evaluation examples. The identified tools shared constructs which were evaluated, including: Stakeholders, Data, Research Evidence, Implementation, and Sociotechnical Infrastructure. However, there was divergence in evaluation methodology. Tools ranged from a five-point numerical rating system for process maturity with a radar chart called the Network Maturity Grid (NMG); the Kaiser Permanente Washington (KPWA) LHS Logic Model, which provides a broad list of constructs and sample measures relevant to LHS operations; and finally LADDERS, a simple tool or form-based template designed for consistent evaluation over time. The NMG tool was the most mature in terms of adaptation and adoption. Notably, two (NMG and the KPWA LHS Logic Model) out of three tools conceptualized the LHS as a suite of processes and devised tools were processes that linked these constructs. IMPLICATIONS FOR TOOLKIT DEVELOPMENT The evaluation of LHS implementation remains an under explored area of investigation, as this scoping review found only three tools for LHS implementation evaluation. Our findings indicate a need for further empirical research in this area and suggest early consensus in constructs that need to be considered during evaluation.
Collapse
Affiliation(s)
- Darren Rajit
- Monash Centre for Health Research and Implementation, Faculty of Medicine, Nursing, and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - Sandra Reeder
- Monash Centre for Health Research and Implementation, Faculty of Medicine, Nursing, and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - Alison Johnson
- Monash Centre for Health Research and Implementation, Faculty of Medicine, Nursing, and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - Joanne Enticott
- Monash Centre for Health Research and Implementation, Faculty of Medicine, Nursing, and Health Sciences, Monash University, Melbourne, VIC, Australia.
- Monash Partners Academic Health Sciences Centre, Melbourne, VIC, Australia.
| | - Helena Teede
- Monash Centre for Health Research and Implementation, Faculty of Medicine, Nursing, and Health Sciences, Monash University, Melbourne, VIC, Australia
- Monash Partners Academic Health Sciences Centre, Melbourne, VIC, Australia
- Monash Health Endocrinology and Diabetes Departments, Melbourne, VIC, Australia
| |
Collapse
|
4
|
Harrison MI, Borsky AE. Funding Learning Health System Research: Challenges and Strategies. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:673-682. [PMID: 38363814 DOI: 10.1097/acm.0000000000005661] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/18/2024]
Abstract
PURPOSE A growing number of health systems are establishing learning health system (LHS) programs, where research focuses on rapidly improving the health system's internal operations and performance. The authors examine funding challenges facing such initiatives and identify strategies for managing tensions between reliance on external research funding and directly contributing to improvement and learning within the researchers' own system. METHOD Qualitative case studies of LHS research programs in 5 health systems were performed via 38 semistructured interviews (October 2019-April 2021) with 35 diverse respondents. Inductive and deductive rapid qualitative analysis supported interview, system-level, and cross-system summaries and analysis. RESULTS External funding awards to LHS researchers facilitated some internal improvement and learning, scientific advancements, and the reputation of researchers and their systems, but reliance on external funding also challenged researchers' responsiveness to concerns of system leaders, managers, practitioners, and system needs. Gaps between external funding requirements and internally focused projects arose in objectives, practical applicability, audiences, timetables, routines, skill sets, and researchers' careers. To contribute more directly to system improvement, LHS researchers needed to collaborate with clinicians and other nonresearchers and pivot between long research studies and shorter, dynamic improvement, evaluation, and data analysis projects. With support from system executives, LHS program leaders employed several strategies to enhance researchers' internal contributions. They aligned funded-research topics with long-term system needs, obtained internal funding for implementing and sustaining practice change, and diversified funding sources. CONCLUSIONS To foster LHS research contributions to internal system learning and improvement, LHS program leaders need to manage tensions between concentrating on externally funded research and fulfilling their mission of providing research-based services to their own system. Health system executives can support LHS programs by setting clear goals for them; appropriately staffing, budgeting, and incentivizing LHS researchers; and developing supportive, system-wide teamwork, skill development programs, and data infrastructures.
Collapse
|
5
|
Van Citters AD, Buus‐Frank ME, King JR, Seid M, Holthoff MM, Amin RS, Britto MT, Nelson EC, Marshall BC, Sabadosa KA. The Cystic Fibrosis Learning Network: A mixed methods evaluation of program goals, attributes, and impact. Learn Health Syst 2023; 7:e10356. [PMID: 37731865 PMCID: PMC10508326 DOI: 10.1002/lrh2.10356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 11/24/2022] [Accepted: 12/05/2022] [Indexed: 12/24/2022] Open
Abstract
Introduction The Cystic Fibrosis (CF) Foundation sponsored the design, pilot testing, and implementation of the CF Learning Network (CFLN) to explore how the Foundation's Care Center Network (CCN) could become a learning health system. Six years after the design, the Foundation commissioned a formative mixed methods evaluation of the CFLN to assess: CFLN participants' understanding of program goals, attributes, and perceptions of current and future impact. Methods We performed semi-structured interviews with CFLN participants to identify perceived goals, attributes, and impact of the network. Following thematic analyses, we developed and distributed a survey to CFLN members and a matched sample of CCN programs to understand whether the themes were unique to the CFLN. Results Interviews with 24 CFLN participants were conducted. Interviewees identified the primary CFLN goal as improving outcomes for people living with CF, with secondary goals of providing training in quality improvement (QI), creating a learning community, engaging all stakeholders in improvement, and spreading best practices to the CCN. Project management, use of data, common QI methods, and the learning community were seen as critical to success. Survey responses were collected from 103 CFLN members and 25 CCN members. The data revealed that CFLN respondents were more likely than CCN respondents to connect with other CF programs, routinely use data for QI, and engage patient and family partners in QI. Conclusions Our study suggests that the CFLN provides value beyond that achieved by the CCN. Key questions remain about whether spread of the CFLN could improve outcomes for more people living with CF.
Collapse
Affiliation(s)
- Aricca D. Van Citters
- The Dartmouth Institute for Health Policy and Clinical PracticeGeisel School of MedicineLebanonNew HampshireUSA
| | - Madge E. Buus‐Frank
- The Dartmouth Institute for Health Policy and Clinical PracticeGeisel School of MedicineLebanonNew HampshireUSA
- Department of PediatricsDartmouth Health Children'sLebanonNew HampshireUSA
| | - Joel R. King
- The Dartmouth Institute for Health Policy and Clinical PracticeGeisel School of MedicineLebanonNew HampshireUSA
| | - Michael Seid
- Division of Pulmonary MedicineCincinnati Children's Hospital Medical Center and the University of Cincinnati College of MedicineCincinnatiOhioUSA
- James M Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOHUSA
| | - Megan M. Holthoff
- The Dartmouth Institute for Health Policy and Clinical PracticeGeisel School of MedicineLebanonNew HampshireUSA
| | - Raouf S. Amin
- Division of Pulmonary MedicineCincinnati Children's Hospital Medical Center and the University of Cincinnati College of MedicineCincinnatiOhioUSA
| | - Maria T. Britto
- James M Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOHUSA
| | - Eugene C. Nelson
- The Dartmouth Institute for Health Policy and Clinical PracticeGeisel School of MedicineLebanonNew HampshireUSA
| | | | | |
Collapse
|
6
|
Ong T, Albon D, Amin RS, Bailey J, Bandla S, Britto MT, Flath J, Gamel B, Powers M, Sabadosa KA, Saulitis AK, Thomas LK, Thurmond S, Seid M. Establishing a Cystic Fibrosis Learning Network: Interventions to promote collaboration and data-driven improvement at scale. Learn Health Syst 2023; 7:e10354. [PMID: 37448461 PMCID: PMC10336485 DOI: 10.1002/lrh2.10354] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 10/16/2022] [Accepted: 10/25/2022] [Indexed: 12/24/2022] Open
Abstract
Introduction A learning health network is a type of learning health system in which stakeholders use network organization to improve health and health care. Building on existing resources in the cystic fibrosis (CF) community, the Cystic Fibrosis Learning Network (CFLN) was designed to improve medical outcomes and quality of life through an intentional focus on achieving reliable evidence-based chronic care delivery and creating a system for data-driven collaborative learning. Methods We describe the development and growth of the CFLN considering six domains of a Network Maturity Grid: system leadership; governance and policy management; quality improvement (QI); engagement and community building; data and analytics; and research. We illustrate the impact of the CFLN experience on chronic care processes and indicators of collaborative infrastructure. Results The CFLN represents 36 accredited care centers in the CF Foundation Care Center Network caring for over 6300 patients. Of 6779 patient clinical care visits/quarter, 77% are entered into the CF Foundation Patient Registry within 30 days, providing timely means to track outcomes. Collaborative visit planning is occurring in 93% of clinical care visits to share agenda setting with patients and families. Almost all CFLN teams (94%, n = 34) have a patient/family partner (PFP), and 74% of PFPs indicate they are actively participating, taking ownership of, or leading QI initiatives with the interdisciplinary care team. In 2022, 97% of centers reported completing 1-13 improvement cycles per month, and 82% contributed to monthly QI progress reports to share learning. Conclusion The CFLN is a maturing, collaborative infrastructure. CFLN centers practice at an advanced level of coproduction. The CFLN fosters interdisciplinary and PFP leadership and the performance of consistent data-driven improvement cycles. CFLN centers are positioned to respond to rapid changes in evidence-based care and advance the practice of QI and implementation science on a broader scale.
Collapse
Affiliation(s)
- Thida Ong
- Division of Pulmonary and Sleep Medicine, Department of PediatricsUniversity of Washington, Seattle Children's HospitalSeattleWashingtonUSA
| | - Dana Albon
- Division of Pulmonary Medicine, Department of Internal MedicineUVACharlottesvilleVirginiaUSA
| | - Raouf S. Amin
- Division of Pulmonary and Sleep Medicine, Department of Pediatrics CincinnatiChildren's Hospital Medical CenterCincinnatiOhioUSA
| | - Julianna Bailey
- Division of Pulmonary, Allergy and Critical Care MedicineUniversity of Alabama at BirminghamBirminghamAlabamaUSA
| | - Srujana Bandla
- James M. Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
| | - Maria T. Britto
- James M. Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
| | - Jonathan Flath
- Cystic Fibrosis Center, Division of Pulmonary, Allergy, Critical Care, and Sleep MedicineUniversity of MinnesotaMinneapolisMinnesotaUSA
| | - Breck Gamel
- Children's Health Pediatric Cystic Fibrosis Center DallasUTSWDallasTexasUSA
| | - Michael Powers
- Pediatrics Doernbecher Children's HospitalOHSUPortlandOregonUSA
| | | | - Anna K. Saulitis
- Adult and Pediatric Cystic Fibrosis Care CentersRush University Medical CenterChicagoIllinoisUSA
| | - Lacrecia K. Thomas
- Cystic Fibrosis Center Children's of AlabamaChildren's of AlabamaBirminghamAlabamaUSA
| | - Sophia Thurmond
- Division of Pulmonary and Sleep Medicine, Department of Pediatrics CincinnatiChildren's Hospital Medical CenterCincinnatiOhioUSA
| | - Michael Seid
- Division of Pulmonary and Sleep Medicine, Department of Pediatrics CincinnatiChildren's Hospital Medical CenterCincinnatiOhioUSA
| | | |
Collapse
|
7
|
Foley T, Vale L. A framework for understanding, designing, developing and evaluating learning health systems. Learn Health Syst 2023; 7:e10315. [PMID: 36654802 PMCID: PMC9835047 DOI: 10.1002/lrh2.10315] [Citation(s) in RCA: 18] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Revised: 04/19/2022] [Accepted: 05/01/2022] [Indexed: 01/21/2023] Open
Abstract
Introduction A Learning Health System is not a technical project. It is the evolution of an existing health system into one capable of learning from every patient. This paper outlines a recently published framework intended to aid the understanding, design, development and evaluation of Learning Health Systems. Methods This work extended an existing repository of Learning Health System evidence, adding five more workshops. The total was subjected to thematic analysis, yielding a framework of elements important to understanding, designing, developing and evaluating Learning Health Systems. Purposeful literature reviews were conducted on each element. The findings were revised following a review by a group of international experts. Results The resulting framework was arranged around four questions:What is our rationale for developing a Learning Health System?There can be many reasons for developing a Learning Health System. Understanding these will guide its development.What sources of complexity exist at the system and the intervention level?An understanding of complexity is central to making Learning Health Systems work. The non-adoption, abandonment, scale-up, spread and sustainability framework was utilised to help understand and manage it.What strategic approaches to change do we need to consider?A range of strategic issues must be addressed to enable successful change in a Learning Health System. These include, strategy, organisational structure, culture, workforce, implementation science, behaviour change, co-design and evaluation.What technical building blocks will we need?A Learning Health System must capture data from practice, turn it into knowledge and apply it back into practice. There are many methods to achieve this and a range of platforms to help. Discussion The results form a framework for understanding, designing, developing and evaluating Learning Health Systems at any scale. Conclusion It is hoped that this framework will evolve with use and feedback.
Collapse
Affiliation(s)
- Tom Foley
- PI Learning Healthcare Project, Health Economics GroupPopulation Health Sciences Institute, Newcastle UniversityNewcastle‐upon‐TyneUK
| | - Luke Vale
- Health Economics GroupPopulation Health Sciences Institute, Newcastle UniversityNewcastle‐upon‐TyneUK
| |
Collapse
|
8
|
Lozano PM, Lane‐Fall M, Franklin PD, Rothman RL, Gonzales R, Ong MK, Gould MK, Beebe TJ, Roumie CL, Guise J, Enders FT, Forrest CB, Mendonca EA, Starrels JL, Sarkar U, Savitz LA, Moon J, Linzer M, Ralston JD, Chesley FD. Training the next generation of learning health system scientists. Learn Health Syst 2022; 6:e10342. [PMID: 36263260 PMCID: PMC9576226 DOI: 10.1002/lrh2.10342] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Revised: 08/15/2022] [Accepted: 08/18/2022] [Indexed: 12/19/2022] Open
Abstract
Introduction The learning health system (LHS) aligns science, informatics, incentives, stakeholders, and culture for continuous improvement and innovation. The Agency for Healthcare Research and Quality and the Patient-Centered Outcomes Research Institute designed a K12 initiative to grow the number of LHS scientists. We describe approaches developed by 11 funded centers of excellence (COEs) to promote partnerships between scholars and health system leaders and to provide mentored research training. Methods Since 2018, the COEs have enlisted faculty, secured institutional resources, partnered with health systems, developed and implemented curricula, recruited scholars, and provided mentored training. Program directors for each COE provided descriptive data on program context, scholar characteristics, stakeholder engagement, scholar experiences with health system partnerships, roles following program completion, and key training challenges. Results To date, the 11 COEs have partnered with health systems to train 110 scholars. Nine (82%) programs partner with a Veterans Affairs health system and 9 (82%) partner with safety net providers. Clinically trained scholars (n = 87; 79%) include 70 physicians and 17 scholars in other clinical disciplines. Non-clinicians (n = 29; 26%) represent diverse fields, dominated by population health sciences. Stakeholder engagement helps scholars understand health system and patient/family needs and priorities, enabling opportunities to conduct embedded research, improve outcomes, and grow skills in translating research methods and findings into practice. Challenges include supporting scholars through roadblocks that threaten to derail projects during their limited program time, ranging from delays in access to data to COVID-19-related impediments and shifts in organizational priorities. Conclusions Four years into this novel training program, there is evidence of scholars' accomplishments, both in traditional academic terms and in terms of moving along career trajectories that hold the potential to lead and accelerate transformational health system change. Future LHS training efforts should focus on sustainability, including organizational support for scholar activities.
Collapse
Affiliation(s)
- Paula M. Lozano
- Kaiser Permanente Washington Health Research InstituteSeattleWashingtonUSA
| | - Meghan Lane‐Fall
- Department of Anesthesiology and Critical CareUniversity of Pennsylvania Perelman School of MedicinePhiladelphiaPennsylvaniaUSA
- Department of Biostatistics, Epidemiology, and InformaticsUniversity of Pennsylvania Perelman School of MedicinePhiladelphiaPennsylvaniaUSA
| | - Patricia D. Franklin
- Department of Medical Social ScienceNorthwestern University Feinberg School of MedicineChicagoIllinoisUSA
| | - Russell L. Rothman
- Institute for Medicine and Public HealthVanderbilt University Medical CenterNashvilleTennesseeUSA
| | - Ralph Gonzales
- Department of Medicine, Division of General Internal MedicineUCSFSan FranciscoCaliforniaUSA
- Continuous Improvement DepartmentUCSF HealthSan FranciscoCaliforniaUSA
| | - Michael K. Ong
- Department of MedicineUCLALos AngelesCaliforniaUSA
- Department of Health Policy and ManagementUCLALos AngelesCaliforniaUSA
- VA Greater Los Angeles Healthcare SystemLos AngelesCaliforniaUSA
| | - Michael K. Gould
- Department of Health System ScienceKaiser Permanente Bernard J. Tyson School of MedicinePasadenaCaliforniaUSA
| | - Timothy J. Beebe
- School of Public HealthUniversity of MinnesotaMinneapolisMinnesotaUSA
| | - Christianne L. Roumie
- Division of General Internal Medicine and Public HealthVanderbilt University Medical CenterNashvilleTennesseeUSA
| | - Jeanne‐Marie Guise
- Department of Obstetrics and GynecologyOHSU‐PSU School of Public HealthPortlandOregonUSA
- Department of Medical Informatics and Clinical EpidemiologyOHSU‐PSU School of Public HealthPortlandOregonUSA
- Department of Emergency MedicineOHSU‐PSU School of Public HealthPortlandOregonUSA
| | - Felicity T. Enders
- Department of Quantitative Health ScienceMayo Clinic College of Medicine and ScienceRochesterMinnesotaUSA
| | - Christopher B. Forrest
- Applied Clinical Research CenterChildren's Hospital of PhiladelphiaPhiladelphiaPennsylvaniaUSA
| | - Eneida A. Mendonca
- Center for Biomedical InformaticsRegenstrief Institute, Inc.IndianapolisIndianaUSA
- Department of PediatricsIndiana University School of MedicineIndianapolisIndianaUSA
- Department of BiostatisticsIndiana University School of MedicineIndianapolisIndianaUSA
| | - Joanna L. Starrels
- Department of MedicineAlbert Einstein College of MedicineBronxNew YorkUSA
| | - Urmimala Sarkar
- UCSF Department of Medicine, Division of General Internal MedicineUCSF Center for Vulnerable Populations, Zuckerberg San Francisco General HospitalSan FranciscoCaliforniaUSA
| | - Lucy A. Savitz
- Kaiser Permanente Center for Health ResearchPortlandOregonUSA
| | - JeanHee Moon
- Applied Clinical Research CenterChildren's Hospital of PhiladelphiaPhiladelphiaPennsylvaniaUSA
| | - Mark Linzer
- Department of Medicine and the Institute for Professional WorklifeHennepin Healthcare and University of Minnesota Medical SchoolMinneapolisMinnesotaUSA
| | - James D. Ralston
- Kaiser Permanente Washington Health Research InstituteSeattleWashingtonUSA
| | | |
Collapse
|
9
|
Trinkley KE, Ho PM, Glasgow RE, Huebschmann AG. How Dissemination and Implementation Science Can Contribute to the Advancement of Learning Health Systems. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1447-1458. [PMID: 35796045 PMCID: PMC9547828 DOI: 10.1097/acm.0000000000004801] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Many health systems are working to become learning health systems (LHSs), which aim to improve the value of health care by rapidly, continuously generating evidence to apply to practice. However, challenges remain to advance toward the aspirational goal of becoming a fully mature LHS. While some important challenges have been well described (i.e., building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data), other key challenges are underrecognized, including balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on external validity and reproducibility (generalizability), and designing for sustainability. Many LHSs focus on continuous learning cycles, but with limited consideration of issues related to the rapidity of these learning cycles, as well as the sustainability or generalizability of solutions. Some types of data have been consistently underrepresented, including patient-reported outcomes and preferences, social determinants, and behavioral and environmental data, the absence of which can exacerbate health disparities. A promising approach to addressing many challenges that LHSs face may be found in dissemination and implementation (D&I) science. With an emphasis on multilevel dynamic contextual factors, representation of implementation partner engagement, pragmatic research, sustainability, and generalizability, D&I science methods can assist in overcoming many of the challenges facing LHSs. In this article, the authors describe the current state of LHSs and challenges to becoming a mature LHS, propose solutions to current challenges, focusing on the contributions of D&I science with other methods, and propose key components and characteristics of a mature LHS model that others can use to plan and develop their LHSs.
Collapse
Affiliation(s)
- Katy E Trinkley
- K.E. Trinkley is associate professor, Departments of Clinical Pharmacy and Medicine and Adult and Child Consortium for Outcomes Research and Delivery Science (ACCORDS), University of Colorado Anschutz Medical Center, and clinical informaticist, Department of Clinical Informatics, UCHealth, Aurora, Colorado; ORCID: http://orcid.org/0000-0003-2041-7404
| | - P Michael Ho
- P.M. Ho is professor, Department of Medicine, University of Colorado Anschutz Medical Campus, and professor, VA Eastern Colorado Health Care System, Aurora, Colorado; ORCID: http://orcid.org/0000-0002-7775-6266
| | - Russell E Glasgow
- R.E. Glasgow is research professor, Department of Family Medicine, and director, Dissemination and Implementation Science Program, ACCORDS, University of Colorado Anschutz Medical Center, Aurora, Colorado; ORCID: http://orcid.org/0000-0003-4218-3231
| | - Amy G Huebschmann
- A.G. Huebschmann is associate professor, Division of General Internal Medicine, ACCORDS and Ludeman Family Center for Women's Health Research, University of Colorado Anschutz Medical Center, Aurora, Colorado; ORCID: http://orcid.org/0000-0002-9329-3142
| |
Collapse
|
10
|
Koscielniak N, Jenkins D, Hassani S, Buckon C, Tucker JS, Sienko S, Tucker CA. The SHOnet learning health system: Infrastructure for continuous learning in pediatric rehabilitation. Learn Health Syst 2022; 6:e10305. [PMID: 35860324 PMCID: PMC9284925 DOI: 10.1002/lrh2.10305] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Revised: 12/12/2021] [Accepted: 01/20/2022] [Indexed: 11/29/2023] Open
Abstract
Introduction To describe the development and implementation of learning health system (LHS) infrastructure for a pediatric specialty care health system to support LHS research in pediatric rehabilitation settings. Methods An existing pediatric common data model (eg, PEDSnet) of standardized medical terminologies for research was expanded and leveraged for this stud, and applied to SHOnet, a clinical research data resource consisting of deidentified data extracted from the electronic health record (EHR) from the Shriners Hospitals for Children speacialty pediatric health care system. We mapped EHR data for laboratory, procedures, drugs, and conditions to standardized vocabularies including ICD-10, CPT, RxNorm, and LOINC to the common data model using an established extraction-transformation-loading process. Rigorous quality checks were conducted to ensure a high degree of data conformance, completeness, and plausibility. SHOnet data elements from all sources are de-identified and the server is managed by the SHC Information Systems Department. SHOnet data are refreshed monthly and data elements are continually expanded based on new research endeavors. Interventions Not applicable. Results The Shriners Health Outcomes Network (SHOnet) includes data for over 10 000 distinct observational data elements based on over two million patient encounters between 2011 and present. Conclusion The systematic process to develop SHOnet is replicable and flexible for other pediatric rehabilitation research settings interested in building out their LHS capabilities. Challenges and facilitators may arise for building such LHS infrastructure for rehabilitation in areas of (a) data capture, curation, query, and governance, (b) generating knowledge from data, and (c) dissemination and implementation of new institutional knowledge. Further research studies are needed to evaluate these data resources for scalable system-learning endeavors.SHOnet is an exemplar of an LHS for rehabilitation and specialty care settings. The success of an LHS is dependent on engagement of multiple stakeholders, shared governance, effective knowledge translation, and deep commitment to long-term strategies for engaging clinicians, administration, and families in leveraging knowledge to improve clinical outcomes.
Collapse
Affiliation(s)
- Nikolas Koscielniak
- Clinical and Translational Science InstituteWake Forest School of MedicineWinston‐SalemNorth CarolinaUSA
| | - Diane Jenkins
- Quality Measurement & Performance ImprovementShriners Hospitals for ChildrenTampaFloridaUSA
| | - Sahar Hassani
- Clinical ResearchShriners Hospitals for ChildrenChicagoIllinoisUSA
| | - Cathleen Buckon
- Clinical ResearchShriners Hospitals for ChildrenPortlandOregonUSA
| | - Joshua S. Tucker
- Department of Biomedical InformaticsChildren's Hospital ColoradoAuroraColoradoUSA
| | - Susan Sienko
- Clinical ResearchShriners Hospitals for ChildrenPortlandOregonUSA
| | - Carole A. Tucker
- Division of Rehabilitation SciencesUniversity of Texas Medical BranchGalvestonTexasUSA
| |
Collapse
|
11
|
Vinson AH, Seid M, Gamel B, Saeed S, Fureman B, Cronin SC, Bates K, Hartley D. Toward an ontology of collaborative learning healthcare systems. Learn Health Syst 2022; 6:e10306. [PMID: 35860315 PMCID: PMC9284927 DOI: 10.1002/lrh2.10306] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2020] [Revised: 11/22/2021] [Accepted: 01/20/2022] [Indexed: 11/13/2022] Open
Abstract
Objective To establish a basis for a domain ontology - a formal, explicit specification of a shared conceptualization - of collaborative learning healthcare systems (CLHSs) in order to facilitate measurement, explanation, and improvement. Methods We adapted the "Methontology" approach to begin building an ontology of CLHSs. We specified the purpose of an ontology, acquired domain knowledge via literature review, conceptualized a common framework of CLHSs using a grounded approach, refined these concepts based on expert panel input, and illustrated concept application via four cases. Results The set of concepts identified as important to include in an ontology includes goals, values, structure, actors, environment, and products. To establish this set of concepts, we gathered input from content experts in two ways. First, expert panel methods were used to elicit feedback on these concepts and to test the elicitation of terms for the vocabulary of the Values concept. Second, from these discussions we developed a mapping exercise to test the intuitiveness of the concepts, requesting that network leaders from four CLHSs complete a mapping exercise to associate characteristics of their networks with the high-level concepts, building the vocabulary for each concept in a grounded fashion. We also solicited feedback from these participants on the experience of completing the mapping exercise, finding that the exercise is acceptable and could aid in CLHS development and collaboration. Respondents identified opportunities to improve the operational definitions of each concept to ensure that corresponding vocabularies are distinct and non-overlapping. Discussion Our results provide a foundation for developing a formal, explicit shared conceptualization of CLHSs. Once developed, such a tool can be useful for measurement, explanation, and improvement. Further work, including alignment to a top-level ontology, expanding the vocabulary, and defining relations between vocabulary is required to formally build out an ontology for these uses.
Collapse
Affiliation(s)
- Alexandra H. Vinson
- Department of Learning Health SciencesUniversity of MichiganAnn ArborMichiganUSA
| | - Michael Seid
- Division of Pulmonary MedicineCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- James M Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Department of PediatricsUniversity of Cincinnati College of MedicineCincinnatiOhioUSA
| | - Breck Gamel
- Co‐Chair, Network Leadership TeamCystic Fibrosis Learning NetworkUSA
| | - Shehzad Saeed
- Department of Medical AffairsDayton Children's HospitalDaytonOhioUSA
| | - Brandy Fureman
- Mission Outcomes TeamEpilepsy FoundationBethesdaMarylandUSA
| | - Susan C. Cronin
- Division of Pulmonary MedicineCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
| | - Katherine Bates
- Department of Pediatrics, C.S. Mott Children's HospitalUniversity of Michigan Medical SchoolAnn ArborMichiganUSA
| | - David Hartley
- James M Anderson Center for Health Systems ExcellenceCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Department of PediatricsUniversity of Cincinnati College of MedicineCincinnatiOhioUSA
| |
Collapse
|
12
|
Ferguson L, Rentes VC, McCarthy L, Vinson AH. Collaborative conversations during the time of COVID-19: Building a "meta"-learning community. Learn Health Syst 2022; 6:e10284. [PMID: 35036555 PMCID: PMC8753305 DOI: 10.1002/lrh2.10284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 04/19/2021] [Accepted: 06/22/2021] [Indexed: 11/26/2022] Open
Abstract
PROBLEM COVID-19 created new research, clinical, educational, and personal challenges, while simultaneously separating work teams who were under work-from-home restrictions. Addressing these challenges required new forms of collaborative groups. APPROACH To support the department community and the rapid sharing of new research, educational, clinical, and personal efforts, a Core Team from the Department of Learning Health Sciences at the University of Michigan developed a meeting series called the COVID Conversations. This Experience Report shares the organizational structure of the COVID Conversations, proposes a comparison to traditional Learning Communities, and reports the results of a questionnaire that gathered details about department members' COVID-related activities. OUTCOMES We identify and describe salient similarities and differences between the COVID Conversations and the characteristics of Learning Communities. We also developed and piloted a taxonomy for characterizing LHS research projects that may be further developed for use in Learning Community planning, in conjunction with other maturity grids and ontologies. We propose the term "Meta-Learning Community" to describe the structure and function of the COVID Conversations. NEXT STEPS In academic medicine, remote work, telemedicine, and virtual learning may be here to stay. The COVID Conversations constitute a distinct and innovative form of collaborative work in which separate teams addressing distinct goals, yet sharing a common passion to tackle the issues brought by the pandemic, are able to share experiences and learn from one other. The challenges of COVID-19 have made evident the need for multiple forms of organizing teamwork, and our study contributes the notion of a "Meta"-Learning Community as a new form of collaborative work.
Collapse
Affiliation(s)
- Lisa Ferguson
- Department of Learning Health SciencesUniversity of MichiganAnn ArborMichiganUSA
| | - Victor C. Rentes
- Department of Learning Health SciencesUniversity of MichiganAnn ArborMichiganUSA
| | - Lauren McCarthy
- Department of Learning Health SciencesUniversity of MichiganAnn ArborMichiganUSA
| | - Alexandra H. Vinson
- Department of Learning Health SciencesUniversity of MichiganAnn ArborMichiganUSA
| |
Collapse
|
13
|
Schleyer T, Williams L, Gottlieb J, Weaver C, Saysana M, Azar J, Sadowski J, Frederick C, Hui S, Kara A, Ruppert L, Zappone S, Bushey M, Grout R, Embi PJ. The Indiana Learning Health System Initiative: Early experience developing a collaborative, regional learning health system. Learn Health Syst 2021; 5:e10281. [PMID: 34277946 PMCID: PMC8278436 DOI: 10.1002/lrh2.10281] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2020] [Revised: 05/30/2021] [Accepted: 06/03/2021] [Indexed: 12/18/2022] Open
Abstract
INTRODUCTION Learning health systems (LHSs) are usually created and maintained by single institutions or healthcare systems. The Indiana Learning Health System Initiative (ILHSI) is a new multi-institutional, collaborative regional LHS initiative led by the Regenstrief Institute (RI) and developed in partnership with five additional organizations: two Indiana-based health systems, two schools at Indiana University, and our state-wide health information exchange. We report our experiences and lessons learned during the initial 2-year phase of developing and implementing the ILHSI. METHODS The initial goals of the ILHSI were to instantiate the concept, establish partnerships, and perform LHS pilot projects to inform expansion. We established shared governance and technical capabilities, conducted a literature review-based and regional environmental scan, and convened key stakeholders to iteratively identify focus areas, and select and implement six initial joint projects. RESULTS The ILHSI successfully collaborated with its partner organizations to establish a foundational governance structure, set goals and strategies, and prioritize projects and training activities. We developed and deployed strategies to effectively use health system and regional HIE infrastructure and minimize information silos, a frequent challenge for multi-organizational LHSs. Successful projects were diverse and included deploying a Fast Healthcare Interoperability Standards (FHIR)-based tool across emergency departments state-wide, analyzing free-text elements of cross-hospital surveys, and developing models to provide clinical decision support based on clinical and social determinants of health. We also experienced organizational challenges, including changes in key leadership personnel and varying levels of engagement with health system partners, which impacted initial ILHSI efforts and structures. Reflecting on these early experiences, we identified lessons learned and next steps. CONCLUSIONS Multi-organizational LHSs can be challenging to develop but present the opportunity to leverage learning across multiple organizations and systems to benefit the general population. Attention to governance decisions, shared goal setting and monitoring, and careful selection of projects are important for early success.
Collapse
Affiliation(s)
- Titus Schleyer
- Center for Biomedical InformaticsRegenstrief Institute, IncIndianapolisIndianaUSA
- Department of MedicineIndiana University School of MedicineIndianapolisIndianaUSA
| | - Linda Williams
- Center for Health Services ResearchRegenstrief Institute, IncIndianapolisIndianaUSA
- Department of NeurologyIndiana University School of MedicineIndianapolisIndianaUSA
- VA HSR&D EXTEND QUERIRichard L. Roudebush VA Medical CenterIndianapolisIndianaUSA
| | - Jonathan Gottlieb
- Department of Health AdministrationUniversity of ProvidenceGreat FallsMontanaUSA
| | - Christopher Weaver
- Department of Emergency MedicineIndiana University School of MedicineIndianapolisIndianaUSA
- Physician AdministrationIndiana University HealthIndianapolisIndianaUSA
| | - Michele Saysana
- Physician AdministrationIndiana University HealthIndianapolisIndianaUSA
- Department of PediatricsIndiana University School of MedicineIndianapolisIndianaUSA
| | - Jose Azar
- Department of MedicineIndiana University School of MedicineIndianapolisIndianaUSA
- Division of Quality and Patient SafetyIndiana University HealthIndianapolisIndianaUSA
| | - Josh Sadowski
- Department of Infection PreventionIndiana University HealthIndianapolisIndianaUSA
| | - Chris Frederick
- AdministrationRegenstrief Institute, IncIndianapolisIndianaUSA
| | - Siu Hui
- Center for Biomedical InformaticsRegenstrief Institute, IncIndianapolisIndianaUSA
- Department of Biostatistics & Health Data ScienceIndiana University School of MedicineIndianapolisIndianaUSA
| | - Areeba Kara
- Department of MedicineIndiana University School of MedicineIndianapolisIndianaUSA
| | - Laura Ruppert
- Center for Biomedical InformaticsRegenstrief Institute, IncIndianapolisIndianaUSA
| | - Sarah Zappone
- Center for Biomedical InformaticsRegenstrief Institute, IncIndianapolisIndianaUSA
| | - Michael Bushey
- Department of PsychiatryIndiana University School of MedicineIndianapolisIndianaUSA
- Department of PsychiatryIndiana University HealthIndianapolisIndianaUSA
| | - Randall Grout
- Center for Biomedical InformaticsRegenstrief Institute, IncIndianapolisIndianaUSA
- Department of PediatricsIndiana University School of MedicineIndianapolisIndianaUSA
- InformaticsEskenazi HealthIndianapolisIndianaUSA
| | - Peter J. Embi
- Department of MedicineIndiana University School of MedicineIndianapolisIndianaUSA
- AdministrationRegenstrief Institute, IncIndianapolisIndianaUSA
- AdministrationIndiana University HealthIndianapolisIndianaUSA
| |
Collapse
|
14
|
Seid M, Hartley DM, Margolis PA. A science of collaborative learning health systems. Learn Health Syst 2021; 5:e10278. [PMID: 34277944 PMCID: PMC8278442 DOI: 10.1002/lrh2.10278] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2020] [Revised: 05/12/2021] [Accepted: 05/13/2021] [Indexed: 11/08/2022] Open
Abstract
INTRODUCTION Improving the U.S. healthcare system and health outcomes is one of the most pressing public health challenges of our time. Previously described Collaborative Learning Health Systems (CLHSs) are a promising approach to outcomes improvement. In order to fully realize this promise, a deeper understanding of this phenomenon is necessary. METHODS We drew on our experience over the past decade with CLHSs as well as qualitative literature review to answer three questions: What kind of phenomena are CLHSs? and what is an appropriate scientific approach? How might we frame CLHSs conceptually? What are potential mechanisms of action? RESULTS CLHSs are complex adaptive systems in which all stakeholders are able to collaborate, at scale, to create and share resources to satisfy a variety of needs. This is accomplished by providing infrastructure and services that enable stakeholders to act on their inherent motivations. This framing has implications for both research and practice. CONCLUSION Articulating this framework and potential mechanisms of action should facilitate research to test and refine hypotheses as well as guide practice to develop and optimize this promising approach to improving healthcare systems.
Collapse
Affiliation(s)
- Michael Seid
- Division of Pulmonary MedicineCincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Cincinnati Children's Hospital, James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Department of PediatricsCollege of Medicine, University of CincinnatiCincinnatiOhioUSA
| | - David M. Hartley
- Cincinnati Children's Hospital, James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Department of PediatricsCollege of Medicine, University of CincinnatiCincinnatiOhioUSA
| | - Peter A. Margolis
- Cincinnati Children's Hospital, James M. Anderson Center for Health Systems Excellence, Cincinnati Children's Hospital Medical CenterCincinnatiOhioUSA
- Department of PediatricsCollege of Medicine, University of CincinnatiCincinnatiOhioUSA
| |
Collapse
|
15
|
Seid M, Bridgeland D, Bridgeland A, Hartley DM. A collaborative learning health system agent-based model: Computational and face validity. Learn Health Syst 2021; 5:e10261. [PMID: 34277939 PMCID: PMC8278449 DOI: 10.1002/lrh2.10261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Revised: 01/21/2021] [Accepted: 01/30/2021] [Indexed: 11/12/2022] Open
Abstract
INTRODUCTION Improving the healthcare system is a major public health challenge. Collaborative learning health systems (CLHS) - network organizations that allow all healthcare stakeholders to collaborate at scale - are a promising response. However, we know little about CLHS mechanisms of actions, nor how to optimize CLHS performance. Agent-based models (ABM) have been used to study a variety of complex systems. We translate the conceptual underpinnings of a CLHS to a computational model and demonstrate initial computational and face validity. METHODS CLHSs are organized to allow stakeholders (patients and families, clinicians, researchers) to collaborate, at scale, in the production and distribution of information, knowledge, and know-how for improvement. We build up a CLHS ABM from a population of patient- and doctor-agents, assign them characteristics, and set them into interaction, resulting in engagement, information, and knowledge to facilitate optimal treatment selection. To assess computational and face validity, we vary a single parameter - the degree to which patients influence other patients - and trace its effects on patient engagement, shared knowledge, and outcomes. RESULTS The CLHS ABM, developed in Python and using the open-source modeling framework Mesa, is delivered as a web application. The model is simulated on a cloud server and the user interface is a web browser using Python and Plotly Dash. Holding all other parameters steady, when patient influence increases, the overall patient population activation increases, leading to an increase in shared knowledge, and higher median patient outcomes. CONCLUSIONS We present the first theoretically-derived computational model of CLHSs, demonstrating initial computational and face validity. These preliminary results suggest that modeling CLHSs using an ABM is feasible and potentially valid. A well-developed and validated computational model of the health system may have profound effects on understanding mechanisms of action, potential intervention targets, and ultimately translation to improved outcomes.
Collapse
Affiliation(s)
- Michael Seid
- Division of Pulmonary MedicineCincinnati Children's HospitalCincinnatiOhioUSA
- James M. Anderson Center for Health Systems ExcellenceCincinnati Children's HospitalCincinnatiOhioUSA
- Department of PediatricsUniversity of Cincinnati, College of MedicineCincinnatiOhioUSA
| | | | | | - David M. Hartley
- James M. Anderson Center for Health Systems ExcellenceCincinnati Children's HospitalCincinnatiOhioUSA
- Department of PediatricsUniversity of Cincinnati, College of MedicineCincinnatiOhioUSA
| |
Collapse
|
16
|
|