How can we recognize continuous quality improvement?

Application of Time-Driven Activity-Based Costing for Head and Neck Microvascular Free Flap Reconstruction

OBJECTIVE

Traditional hospital accounting fails to provide an accurate cost of complex surgical care. Here we describe the application of time-driven activity-based costing (TDABC) to characterize costs of head and neck oncologic procedures involving free tissue transfer.

STUDY DESIGN

Retrospective cohort study.

SETTING

Single tertiary academic medical center.

METHODS

An analysis of head and neck oncologic procedures involving microvascular free flap reconstruction from 2018 to 2020 (n = 485) was performed using TDABC methodology to measure cost across operative case and postoperative admission, using quantity of time and cost per unit of each resource to characterize resource utilization. Univariate and generalized linear mixed models were used to examine associations between patient and hospital characteristics and cost of care delivery.

RESULTS

The total cost of care delivery was $41,905.77 ± 21,870.27 with operating room (OR) supplies accounting for only 10% of the total cost. Multivariable analyses identified significant cost drivers including operative time, postoperative length of stay, number of return trips to the OR, postoperative complication, number of free flaps performed, and patient transfer from another hospital or via emergency department admission (P < .05).

CONCLUSION

Operative time and postoperative length of stay, but not operative supplies, were primary drivers of cost of care for head and neck oncology cases involving free tissue transfer. TDABC offers granular cost characterization to inform cost optimization through unused capacity identification and postoperative admission efficiencies.

Barriers and Challenges to Implementing a Quality Improvement Program: Political and Administrative Challenges

Quality improvement (QI) programs have rapidly grown in health care over recent years. Despite increasing evidence of successful QI initiatives resulting in improved outcomes, the adoption and implementation of QI programs remain a challenge worldwide. This paper briefly describes political and administrative barriers that impede the implementation of QI programs, including political and ideological factors, socioeconomic and educational barriers, and barriers related to data collection, privacy, and security. Key political and administrative barriers identified include resource limitations due to inadequate public funding, stringent laws, and change resistance. Potential solutions include support and commitment from regional and national authorities, consultation of all involved parties during QI program development, and financial incentives. The barrier of limited resources is starker among low- and middle-income countries (LMICs) compared with high-income countries (HICs) due to the absence of adequate infrastructure, personnel equipped with QI-oriented skills, and analytical technology. Solutions that have facilitated QI programs in some LMICs include outreach and collaboration with other health centers and established QI programs in HICs. The lack of QI-specific training and education in medical curricula challenges QI implementation but can be mitigated through the provision of QI promotion webinars, QI-specific project opportunities, and formalized QI training modules. Finally, barriers related to data collection, privacy, and security include laws hindering the availability of quality data, inefficient data collection and processes, and outdated clinical information systems. Access to high-quality data, organized record-keeping, and alignment of data collection processes will help alleviate these barriers to QI program implementation. The multidimensional nature of these barriers means that proposed solutions will require coordination from multiple stakeholders, government support, and leaders across multiple fields.

A scoping review of continuous quality improvement in healthcare system: conceptualization, models and tools, barriers and facilitators, and impact

BACKGROUND

The growing adoption of continuous quality improvement (CQI) initiatives in healthcare has generated a surge in research interest to gain a deeper understanding of CQI. However, comprehensive evidence regarding the diverse facets of CQI in healthcare has been limited. Our review sought to comprehensively grasp the conceptualization and principles of CQI, explore existing models and tools, analyze barriers and facilitators, and investigate its overall impacts.

METHODS

This qualitative scoping review was conducted using Arksey and O'Malley's methodological framework. We searched articles in PubMed, Web of Science, Scopus, and EMBASE databases. In addition, we accessed articles from Google Scholar. We used mixed-method analysis, including qualitative content analysis and quantitative descriptive for quantitative findings to summarize findings and PRISMA extension for scoping reviews (PRISMA-ScR) framework to report the overall works.

RESULTS

A total of 87 articles, which covered 14 CQI models, were included in the review. While 19 tools were used for CQI models and initiatives, Plan-Do-Study/Check-Act cycle was the commonly employed model to understand the CQI implementation process. The main reported purposes of using CQI, as its positive impact, are to improve the structure of the health system (e.g., leadership, health workforce, health technology use, supplies, and costs), enhance healthcare delivery processes and outputs (e.g., care coordination and linkages, satisfaction, accessibility, continuity of care, safety, and efficiency), and improve treatment outcome (reduce morbidity and mortality). The implementation of CQI is not without challenges. There are cultural (i.e., resistance/reluctance to quality-focused culture and fear of blame or punishment), technical, structural (related to organizational structure, processes, and systems), and strategic (inadequate planning and inappropriate goals) related barriers that were commonly reported during the implementation of CQI.

CONCLUSIONS

Implementing CQI initiatives necessitates thoroughly comprehending key principles such as teamwork and timeline. To effectively address challenges, it's crucial to identify obstacles and implement optimal interventions proactively. Healthcare professionals and leaders need to be mentally equipped and cognizant of the significant role CQI initiatives play in achieving purposes for quality of care.

Enhancing capability for continuous organisational improvement and learning in healthcare organisations: a systematic review of the literature 2013-2022

BACKGROUND

Healthcare organisations strive to meet their current and future challenges and need to increase their capacity for continuous organisational improvement and learning (COIL). A key aspect of this capacity is the development of COIL capability among employees.

OBJECTIVE

This systematic review aims to explore common attributes of interventions that contribute to the development of COIL capability in healthcare organisations and to explore possible facilitating and hindering factors.

METHODS

A comprehensive search was conducted in Scopus, MEDLINE and Business Source Complete for primary research studies in English or Swedish, in peer-reviewed journals, focusing on organisational improvements and learning in healthcare organisations. Studies were included if they were published between 2013 and 23 November 2022, reported outcomes on COIL capability, included organisations or groups, and were conducted in high-income countries. The included articles were analysed to identify themes related to successful interventions and factors influencing COIL capability.

RESULTS

Thirty-six articles were included, with two studies reporting unsuccessful attempts at increasing COIL capability. The studies were conducted in nine different countries, encompassing diverse units, with the timeframes varying from 15 weeks to 8 years, and they employed quantitative (n=10), qualitative (n=11) and mixed methods (n=15). Analysis of the included articles identified four themes for both attributes of interventions and the factors that facilitated or hindered successful interventions: (1) engaged managers with a strategic approach, (2) external training and guidance to develop internal knowledge, skills and confidence, (3) process and structure to achieve improvements and learning and (4) individuals and teams with autonomy, accountability, and safety.

CONCLUSION

This review provides insights into the intervention attributes that are associated with increasing COIL capability in healthcare organisations as well as factors that can have hindering or facilitating effects. Strategic management, external support, structured processes and empowered teams emerged as key elements for enhancing COIL capability.

Decision-maker roles in healthcare quality improvement projects: a scoping review

OBJECTIVES

Evidence suggests that healthcare quality improvement (QI) projects are more successful when decision-makers are involved in the process. However, guidance regarding the engagement of decision-makers in QI projects is lacking. We conducted a scoping review to identify QI projects involving decision-makers published in the literature and to describe the roles decision-makers played.

METHODS

Following the Joanna Briggs Institute framework for scoping reviews, we systematically searched for all types of studies in English or French between 2002 and 2023 in: EMBASE, MEDLINE via PubMed, PsycINFO, and the Cumulative Index to Nursing and Allied Health Literature. Criteria for inclusion consisted of literature describing health sector QI projects that involved local, regional or system-level decision-makers. Descriptive analysis was performed. Drawing on QI and participatory research literature, the research team developed an inductive data extraction grid to provide a portrait of QI project characteristics, decision-makers' contributions, and advantages and challenges associated with their involvement.

RESULTS

After screening and review, we retained 29 references. 18 references described multi-site projects and 11 were conducted in single sites. Local decision-makers' contributions were documented in 27 of the 29 references and regional decision-makers' contributions were documented in 12. Local decision-makers were more often active participants in QI processes, contributing toward planning, implementation, change management and capacity building. Regional decision-makers more often served as initiators and supporters of QI projects, contributing toward strategic planning, recruitment, delegation, coordination of local teams, as well as assessment and capacity building. Advantages of decision-maker involvement described in the retained references include mutual learning, frontline staff buy-in, accountability, resource allocation, effective leadership and improved implementation feasibility. Considerations regarding their involvement included time constraints, variable supervisory expertise, issues concerning centralised leadership, relationship strengthening among stakeholders, and strategic alignment of frontline staff and managerial priorities CONCLUSIONS: This scoping review provides important insights into the various roles played by decision-makers, the benefits and challenges associated with their involvement, and identifies opportunities for strengthening their engagement. The results of this review highlight the need for practical collaboration and communication strategies that foster partnership between frontline staff and decision-makers at all levels.

Caring for high-need patients

OBJECTIVE

We aimed to explore the construct of "high need" and identify common need domains among high-need patients, their care professionals, and healthcare organizations; and to describe the interventions that health care systems use to address these needs, including exploring the potential unintended consequences of interventions.

METHODS

We conducted a modified Delphi panel informed by an environmental scan. Expert stakeholders included patients, interdisciplinary healthcare practitioners (physicians, social workers, peer navigators), implementation scientists, and policy makers. The environmental scan used a rapid literature review and semi-structured interviews with key informants who provide healthcare for high-need patients. We convened a day-long virtual panel meeting, preceded and followed by online surveys to establish consensus.

RESULTS

The environmental scan identified 46 systematic reviews on high-need patients, 19 empirical studies documenting needs, 14 intervention taxonomies, and 9 studies providing construct validity for the concept "high need." Panelists explored the construct and terminology and established that individual patients' needs are unique, but areas of commonality exist across all high-need patients. Panelists agreed on 11 domains describing patient (e.g., social circumstances), 5 care professional (e.g., communication), and 8 organizational (e.g., staffing arrangements) needs. Panelists developed a taxonomy of interventions with 15 categories (e.g., care navigation, care coordination, identification and monitoring) directed at patients, care professionals, or the organization. The project identified potentially unintended consequences of interventions for high-need patients, including high costs incurred for patients, increased time and effort for care professionals, and identification of needs without resources to respond appropriately.

CONCLUSIONS

Care for high-need patients requires a thoughtful approach; differentiating need domains provides multiple entry points for interventions directed at patients, care professionals, and organizations. Implementation efforts should consider outlined intended and unintended downstream effects on patients, care professionals, and organizations.

FORWARD: Building a Model to Hold Schools of Public Health Accountable for Antiracism Work

Schools of public health have increasingly adopted programs, praxis, and competencies for antiracist work. Fighting Oppression, Racism and White Supremacy through Action, Research and Discourse (FORWARD) was founded to accelerate antiracist work at the Columbia University Mailman School of Public Health in New York City. Seven action corps reporting to an accountability cabinet were established with 183 participants. FORWARD achieved progress across five core pillars. We describe how an iterative, dynamic structure and explicit framework for accountability can guide future antiracism work. (Am J Public Health. 2023;113(10):1086-1088. https://doi.org/10.2105/AJPH.2023.307356).

The Use of Quality Improvement in the Physical Therapy Literature: A Scoping Review

INTRODUCTION

Quality improvement (QI) is a useful methodology for improving healthcare, often through iterative changes. There is no prior review on the application of QI in physical therapy (PT).

PURPOSE AND RELEVANCE

To characterize and evaluate the quality of the QI literature in PT.

METHODS

We searched four electronic databases from inception through September 1, 2022. Included publications focused on QI and included the practice of PT. Quality was assessed using the 16-point QI Minimum Quality Criteria Set (QI-MQCS) appraisal tool.

RESULTS

Seventy studies were included in the review, 60 of which were published since 2014 with most ( n = 47) from the United States. Acute care ( n = 41) was the most prevalent practice setting. Twenty-two studies (31%) did not use QI models or approaches and only nine studies referenced Revised Standards for QI Reporting Excellence guidelines. The median QI-MQCS score was 12 (range 7-15).

CONCLUSIONS/IMPLICATIONS

Quality improvement publications in the PT literature are increasing, yet there is a paucity of QI studies pertaining to most practice settings and a lack of rigor in project design and reporting. Many studies were of low-to-moderate quality and did not meet minimum reporting standards. We recommend use of models, frameworks, and reporting guidelines to improve methodologic rigor and reporting.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs

BACKGROUND

Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions.

METHODS

Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses.

RESULTS

687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame.

CONCLUSIONS

Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

Improving curriculum delivery: Using a results informed quality improvement model for teen behavioral health education

Adolescence is a critical developmental stage to establish healthy decision-making processes and behavior patterns. Many interventions such as evidence-based curricula have been implemented to guide adolescents to avoid risk-taking behaviors and improve health and medical knowledge and outcomes. This study presents a participatory approach informed by the three-stage (3S) quality improvement process model to improve the quality of curriculum delivery, based on the results indicating outcomes achieved, needs for improvement, and quality assurance for maintaining the expected outcomes of an evidence-based curricula. Tests were conducted before and after the intervention. Using threshold levels and measures of change in the tests, instructors participated in guided discussion and analysis of the data to identify where and how instructional improvements should be made and where outcomes were being achieved as expected. This method was used to diagnose variation in the results and delivery and identify root causes informing actions to improve curriculum delivery and outcomes. After the facilitated discussions, pre- and post-tests from subsequent classes were analyzed. The results showed improved test item scores ranging from 2 to 69.5% and seven of 18 items obtained statistical significance following the implementation of the model described. Overall, an increase in the mean percent correct of 17.1% was found.

A scoping review to inform the use of continuous quality improvement in Australian Aboriginal oral health care

ObjectiveThe need to improve existing services to Aboriginal communities is prioritised by Australia's National Oral Health Plan. Although only an emerging area in dentistry, continuous quality improvement (CQI) approaches have positively impacted the delivery of primary health services to Aboriginal communities. This scoping review maps the applicability of CQI strategies to Aboriginal Australian oral healthcare services.MethodsA scoping review was conducted and studies that reported using CQI approaches to improve existing oral health services or quality of care deemed relevant to Aboriginal Australian communities were included.ResultsA total of 73 articles were retrieved and eight articles were included in the final synthesis. Several CQI tools were identified, including: plan-do-study-act cycles, dental quality alliance measures, prioritisation matrices, causal mapping and the use of collective impact methodology.ConclusionData exploring CQI in the context of Aboriginal oral health is scarce. The plan-do-study-act cycle and its variations show potential applicability to Aboriginal oral health care. However, for CQI approaches to be adequately implemented, the prevailing model of dental care requires a paradigm shift from quality assurance to quality improvement, acknowledging the impact of structural and process elements on care.

Disseminating online parenting resources in the community during the COVID-19 pandemic: Lessons learned

Online parenting programs are an effective way to teach behavioral management skills to parents in the absence of in-person resources. This community-engaged study aimed to examine strategies for disseminating online parenting resources in schools. Online resources were disseminated to parents in a Northern California school district. Dissemination strategies were informed by conversations with school principals, teachers, and parents and considered agent, message, and format. A total of 685 parents and teachers clicked on the online resources: 151 parents and 114 teachers attended synchronous classes. The use of dissemination strategies had a compounding influence on the number of synchronous class attendees and clicks. Emails sent by the school district yielded the greatest number of clicks, which was influenced by message content and format. A community-academic partnership (CAP) led to the dissemination of evidence-based online parenting resources to a large population and led to lessons learned that could inform future research involving CAPs.

Effects of service-wide support on regularity of alcohol screening of clients in Australian Aboriginal and Torres Strait Islander Community Controlled Health Services: a cluster randomised trial

Background

We have previously shown that service-wide support can increase the odds of alcohol screening in any 2-month period in a cluster randomized trial of service-wide support to Aboriginal and Torres Strait Islander Community Controlled Health Services (ACCHS). Here we report an exploratory analysis on whether the resulting pattern of screening was appropriate. Aim: we assess whether that increase in screening was associated with: (i) increased first-time screening, (ii) increased annual screening, (iii) whether frequently screened clients fell into one of four risk categories as defined by national guidelines.

Methods

Setting and participants: 22 ACCHS; randomized to receive the support model in the treatment (‘early-support’) arm over 24-months or to the waitlist control arm. Intervention: eight-component support, including training, sharing of experience, audit-and-feedback and resource support. Analysis: records of clients with visits before and after start of implementation were included. Multilevel logistic modelling was used to compare (i) the odds of previously unscreened clients receiving an AUDIT-C screen, (ii) odds of clients being screened with AUDIT-C at least once annually. We describe the characteristics of a sub-cohort of clients who received four or more screens annually, including if they were in a high-risk category.

Results

Of the original trial sample, 43,054 met inclusion criteria, accounting for 81.7% of the screening events in the overall trial. The support did not significantly increase the odds of first-time screening (OR  = 1.33, 95% CI 0.81–2.18, p  = 0.25) or of annual screening (OR  = 0.99, 95% CI 0.42–2.37, p = 0.98). Screening more than once annually occurred in 6240 clients. Of the 841 clients with four or more screens annually, over 50% did not fall into a high-risk category. Females were overrepresented. More males than females fell into high-risk categories.

Conclusion

The significant increase in odds of screening observed in the main trial did not translate to significant improvement in first-time or annual screening following implementation of support. This appeared to be due to some clients being screened more frequently than annually, while more than half remained unscreened. Further strategies to improve alcohol screening should focus on appropriate screening regularity as well as overall rates, to ensure clinically useful information about alcohol consumption.

Trial Registration ACTRN12618001892202, retrospectively registered 16 November 2018 https://anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12618001892202.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13722-022-00294-6.

Ten Years (2011-2021) of the Italian Lombardy ADHD Register for the Diagnosis and Treatment of Children and Adolescents with ADHD

BACKGROUND

The purpose of this article is to update the diagnostic assessment, therapeutic approach, and 12-18 month follow-up of patients added to the Italian Lombardy Attention Deficit Hyperactivity Disorder (ADHD) Register.

METHODS

Medical records of patients added to the Registry from 2011 to 2021 were analysed.

RESULTS

4091 of 5934 patients met the criteria for a diagnosis of ADHD, and 20.3% of them presented a familiarity with the disorder. A total of 2879 children (70.4%) had at least one comorbidity disorder, in prevalence a learning disorder (39%). Nearly all (95.9%) received at least one psychological prescription, 17.9% of them almost one pharmacological treatment, and 15.6% a combination of both. Values of ≥5 of the Clinical Global Impression-Severity scale (CGI-S) are more commonly presented by patients with a pharmacological prescription than with a psychological treatment (p < 0.0001). A significant improvement was reported in half of the patients followed after 1 year, with Clinical Global Impression-Improvement scale (CGI-I) ≤ 3. In all, 233 of 4091 are 18-year-old patients.

CONCLUSIONS

A ten-year systematic monitoring of models of care was a fruitful shared and collaborative initiative in order to promote significant improvement in clinical practice, providing effective and continuous quality of care. The unique experience reported here should spread.

Rating the seriousness of maternal and child health outcomes linked with pregnancy weight gain

BACKGROUND

Current pregnancy weight gain guidelines were developed based on implicit assumptions of a small group of experts about the relative seriousness of adverse health outcomes. Therefore, they will not necessarily reflect the values of women.

OBJECTIVE

To estimate the seriousness of 11 maternal and child health outcomes that have been consistently associated with pregnancy weight gain by engaging patients and health professionals.

METHODS

We collected data using an online panel approach with a modified Delphi structure. We selected a purposeful sample of maternal and child health professionals (n = 84) and women who were pregnant or recently postpartum (patients) (n = 82) in the United States as panellists. We conducted three concurrent panels: professionals only, patients only, and patients and professionals. During a 3-round online modified Delphi process, participants rated the seriousness of health outcomes (Round 1), reviewed and discussed the initial results (Round 2), and revised their original ratings (Round 3). Panellists assigned seriousness ratings (0, [not serious] to 100 [most serious]) for infant death, stillbirth, preterm birth, gestational diabetes, preeclampsia, small-for-gestational-age (SGA) birth, large-for-gestational-age (LGA) birth, unplanned caesarean delivery, maternal obesity, childhood obesity, and maternal metabolic syndrome.

RESULTS

Each panel individually came to a consensus on all seriousness ratings. The final median seriousness ratings combined across all panels were highest for infant death (100), stillbirth (95), preterm birth (80), and preeclampsia (80). Obesity in children, metabolic syndrome in women, obesity in women, and gestational diabetes had median seriousness ratings ranging from 55 to 65. The lowest seriousness ratings were for SGA birth, LGA birth, and unplanned caesarean delivery (30-40).

CONCLUSION

Professionals and women rate some adverse outcomes as being more serious than others. These ratings can be used to establish the range of pregnancy weight gain associated with the lowest risk of a broad range of maternal and child health outcomes.

Measuring the continuous quality improvement orientation of medical education programs

PURPOSE

There is a growing interest in applying continuous quality improvement (CQI) methodologies and tools to medical education contexts. One such tool, the "Are We Making Progress" questionnaire from the Malcolm Baldrige National Quality Award framework, adequately captures the dimensions critical for performance excellence and allows organizations to assess their performance and identify areas for improvement. Its results have been widely validated in business, education, and health care and might be applicable in medical education contexts. The measurement properties of the questionnaire data were analyzed using Rasch modeling to determine if validity evidence, based on Messick's framework, supports the interpretation of results in medical education contexts. Rasch modeling was performed since the questionnaire uses Likert-type scales whose estimates might not be amenable to parametric statistical analyses.

DESIGN/METHODOLOGY/APPROACH

Leaders and teachers at 16 of the 17 Canadian medical schools were invited in 2015-2016 to complete the 40-item questionnaire. Data were analyzed using the ConQuest Rasch calibration program, rating scale model.

FINDINGS

491 faculty members from 11 (69 percent) schools participated. A seven-dimensional, four-point response scale model better fit the data. Overall data fit to model requirements supported the use of person measures with parametric statistics. The structural, content, generalizability, and substantive validity evidence supported the interpretation of results in medical education contexts.

ORIGINALITY/VALUE

For the first time, the Baldrige questionnaire results were validated in medical education contexts. Medical education leaders are encouraged to serially use this questionnaire to measure progress on their school's CQI focus.

Learning and improving in long-term care internationally: the International Prevalence Measurement of Care Quality (LPZ)

Long-term care (LTC) for older adults is an essential part of how health and social care systems respond to population ageing. Different long-term care systems in different countries have taken differing approaches to quality assurance, ranging from inspection-based regulatory systems to data and reporting-based regulatory systems. The significant variability in the ability of long-term care facilities to respond to the COVID-19 pandemic has led to increased recognition of the role of standardized data in informing structured approaches to quality assurance. The International Prevalence Measurement of Care Quality (in Dutch: Landelijke Prevalentiemeting Zorgkwaliteit – LPZ) was developed to guide continuous quality improvement in long-term care facilities. This special article describes the LPZ tool, developed to provide input for the learning and improvement cycle of multidisciplinary teams in the LTC sector and to help improve care quality.

Increasing Vaginal Chlamydia Trachomatis Testing in Adolescent and Young Adults

OBJECTIVE

The Centers for Disease Control and Prevention recommend testing for Chlamydia trachomatis in sexually active female patients <25 years old using nucleic-acid amplification tests (NAAT) from a vaginal swab. Our providers were typically testing using the less sensitive urine NAATs. We aimed to increase the percentage of urogenital C trachomatis NAATs performed by using vaginal swabs in adolescent female patients ages 10 through 20 years from 1.4% to 25%.

METHODS

We implemented 3 interventions at 3 pediatric practices over 12 months including education, process standardization, and cross-training. We used statistical process control to analyze the effect of interventions on our primary outcome: the percentage of urogenital C trachomatis tests performed with a vaginal swab. Our balance measure was the total number of urogenital C trachomatis tests.

RESULTS

There were 818 urogenital C trachomatis tests performed: 289 before and 529 after the first intervention. Of urogenital C trachomatis tests in the preintervention time period, 1.4% were performed by using vaginal swabs. We surpassed our aim of 25% 6 weeks after the first intervention. We noted sustained improvement after the second intervention, with an average of 68.3% of tests performed by using vaginal swabs for the remaining postintervention period. There was no difference in the overall number of urogenital C trachomatis tests pre- and postintervention.

CONCLUSIONS

Using quality improvement methodology and implementing easily replicable interventions, we significantly and sustainably increased use of vaginal swabs. The interventions standardizing processes were associated with a higher impact than the educational intervention.

Engaging Patients and Professionals to Evaluate the Seriousness of Maternal and Child Health Outcomes: Protocol for a Modified Delphi Study

BACKGROUND

Maternal weight gain during pregnancy is one of the few potentially modifiable risk factors for many adverse maternal and child health outcomes. Defining the optimal pregnancy weight gain range is difficult because, while lower weight gain may prevent some outcomes, such as maternal and child obesity, it may increase the risk of others such as fetal growth restriction and infant death. These health outcomes vary in their seriousness to mothers and their health care providers, and these differences in seriousness should be taken into account when determining optimal weight gain ranges. However, the relative seriousness that women and their care providers place on different health outcomes is unknown.

OBJECTIVE

We will determine the seriousness of 11 maternal and child health outcomes that have been consistently associated with pregnancy weight gain. We will achieve this by engaging patients and maternal and child health professionals using an online modified Delphi panel process.

METHODS

We aim to recruit a racially/ethnically and geographically diverse group of 90 US maternal and child health professionals and 90 women who are pregnant or less than 2 years postpartum. We will conduct 3 concurrent panels using the ExpertLens system, a previously evaluated online modified Delphi system that combines 2 rounds of rating with 1 round of feedback and moderated online discussion. In Round 1, panelists are asked to rate the seriousness of each health outcome on a scale of 0-100 and to provide a rationale for their scores. In Round 2, panelists will review their responses relative to those of other panelists. They will discuss their seriousness ratings anonymously using a moderated online discussion board. In Round 3, participants will revise their Round 1 responses based on group feedback and discussion. Each round will be open for 1-2 weeks.

RESULTS

The study protocol was reviewed by our ethics boards and did not require approval as human research. A pilot study of 6 professionals and 7 patients was completed in December 2019.

CONCLUSIONS

Our numeric estimates of the seriousness of maternal and child health outcomes will enable future studies to determine pregnancy weight gain ranges that balance the risks of low and high weight gain for mothers and children.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/16478.

The effectiveness of continuous quality improvement for developing professional practice and improving health care outcomes: a systematic review

BACKGROUND

Efforts to improve the quality, safety, and efficiency of health care provision have often focused on changing approaches to the way services are organized and delivered. Continuous quality improvement (CQI), an approach used extensively in industrial and manufacturing sectors, has been used in the health sector. Despite the attention given to CQI, uncertainties remain as to its effectiveness given the complex and diverse nature of health systems. This review assesses the effectiveness of CQI across different health care settings, investigating the importance of different components of the approach.

METHODS

We searched 11 electronic databases: MEDLINE, CINAHL, EMBASE, AMED, Academic Search Complete, HMIC, Web of Science, PsycINFO, Cochrane Central Register of Controlled Trials, LISTA, and NHS EED to February 2019. Also, we searched reference lists of included studies and systematic reviews, as well as checking published protocols for linked papers. We selected randomized controlled trials (RCTs) within health care settings involving teams of health professionals, evaluating the effectiveness of CQI. Comparators included current usual practice or different strategies to manage organizational change. Outcomes were health care professional performance or patient outcomes. Studies were published in English.

RESULTS

Twenty-eight RCTs assessed the effectiveness of different approaches to CQI with a non-CQI comparator in various settings, with interventions differing in terms of the approaches used, their duration, meetings held, people involved, and training provided. All RCTs were considered at risk of bias, undermining their results. Findings suggested that the benefits of CQI compared to a non-CQI comparator on clinical process, patient, and other outcomes were limited, with less than half of RCTs showing any effect. Where benefits were evident, it was usually on clinical process measures, with the model used (i.e., Plan-Do-Study-Act, Model of Improvement), the meeting type (i.e., involving leaders discussing implementation) and their frequency (i.e., weekly) having an effect. None considered socio-economic health inequalities.

CONCLUSIONS

Current evidence suggests the benefits of CQI in improving health care are uncertain, reflecting both the poor quality of evaluations and the complexities of health services themselves. Further mixed-methods evaluations are needed to understand how the health service can use this proven approach.

TRIAL REGISTRATION

Protocol registered on PROSPERO (CRD42018088309).

A systematic review of approaches to improve practice, detection and treatment of unhealthy alcohol use in primary health care: a role for continuous quality improvement

BACKGROUND

Unhealthy alcohol use involves a spectrum from hazardous use (exceeding guidelines but no harms) through to alcohol dependence. Evidence-based management of unhealthy alcohol use in primary health care has been recommended since 1979. However, sustained and systematic implementation has proven challenging. The Continuing Quality Improvement (CQI) process is designed to enable services to detect barriers, then devise and implement changes, resulting in service improvements.

METHODS

We conducted a systematic review of literature reporting on strategies to improve implementation of screening and interventions for unhealthy alcohol use in primary care (MEDLINE EMBASE, PsycINFO, CINAHL, the Australian Indigenous Health InfoNet). Additional inclusion criteria were: (1) pragmatic setting; (2) reporting original data; (3) quantitative outcomes related to provision of service or change in practice. We investigate the extent to which the three essential elements of CQI are being used (data-guided activities, considering local conditions; iterative development). We compare characteristics of programs that include these three elements with those that do not. We describe the types, organizational levels (e.g. health service, practice, clinician), duration of strategies, and their outcomes.

RESULTS

Fifty-six papers representing 45 projects were included. Of these, 24 papers were randomized controlled trials, 12 controlled studies and 20 before/after and other designs. Most reported on strategies for improving implementation of screening and brief intervention. Only six addressed relapse prevention pharmacotherapies. Only five reported on patient outcomes and none showed significant improvement. The three essential CQI elements were clearly identifiable in 12 reports. More studies with three essential CQI elements had implementation and follow-up durations above the median; utilised multifaceted designs; targeted both practice and health system levels; improved screening and brief intervention than studies without the CQI elements.

CONCLUSION

Utilizing CQI methods in implementation research would appear to be well-suited to drive improvements in service delivery for unhealthy alcohol use. However, the body of literature describing such studies is still small. More well-designed research, including hybrid studies of both implementation and patient outcomes, will be needed to draw clearer conclusions on the optimal approach for implementing screening and treatment for unhealthy alcohol use. (PROSPERO registration ID: CRD42018110475).

Quality improvement in medical schools: vision meets culture

CONTEXT

Medical schools face growing pressures to develop quality improvement (QI) strategies to ensure the continuous quality of their education. To best achieve quality, both organisational processes and culture need to be oriented towards quality. Quality processes already exist at medical schools, at least externally driven by accreditation. However, the dominant culture in most medical schools is not typically oriented towards quality.

OBJECTIVES

This study explores whether QI practices are recognised as such in organisations not culturally QI-oriented. Specifically, it examines faculty members' perceptions about the degree of QI implementation in their medical education programmes. Understanding this perception will inform medical school leadership on how best to use resources for a successful execution of the school's QI vision.

METHODS

Leaders, clinical teachers and formal teachers at 16 of the 17 Canadian medical schools were invited to complete the 'Are We Making Progress?' questionnaire of the Malcolm Baldrige National Quality Award framework, the results of which have been broadly validated. The questionnaire measures the perceived level of QI implementation within organisations using 40 statements grouped under the framework's seven domains of performance excellence.

RESULTS

A total of 491 respondents from 11 (69%) schools completed the questionnaire; 173 (35%) identified as clinical teachers, 150 (31%) as formal teachers, and 168 (34%) as leaders. Perceived QI implementation levels were low across programmes (0.70-1.90 in Rasch person measures) and for each category of respondents. This was especially true for the domains of 'Strategy', 'Measurement/analysis/knowledge management' and 'Operations'. Leaders' perceptions of QI implementation were higher than those of teachers.

CONCLUSIONS

Medical schools' existing QI processes are not recognised as QI activities. For QI strategies to succeed, a programme's culture must embrace QI. In the execution of their QI visions, medical schools should spend resources on embedding quality in the organisation culture in addition to strengthening existing QI practices, especially in the domains listed above.

Spread tools: a systematic review of components, uptake, and effectiveness of quality improvement toolkits

BACKGROUND

The objective was to conduct a systematic review of toolkit evaluations intended to spread interventions to improve healthcare quality. We aimed to determine the components, uptake, and effectiveness of publicly available toolkits.

METHODS

We searched PubMed, CINAHL, and the Web of Science from 2005 to May 2018 for evaluations of publicly available toolkits, used a forward search of known toolkits, screened references, and contacted topic experts. Two independent reviewers screened publications for inclusion. One reviewer abstracted data and appraised the studies, checked by a second reviewer; reviewers resolved disagreements through discussion. Findings, summarized in comprehensive evidence tables and narrative synthesis addressed the uptake and utility, procedural and organizational outcomes, provider outcomes, and patient outcomes.

RESULTS

In total, 77 studies evaluating 72 toolkits met inclusion criteria. Toolkits addressed a variety of quality improvement approaches and focused on clinical topics such as weight management, fall prevention, vaccination, hospital-acquired infections, pain management, and patient safety. Most toolkits included introductory and implementation material (e.g., research summaries) and healthcare provider tools (e.g., care plans), and two-thirds included material for patients (e.g., information leaflets). Pre-post studies were most common (55%); 10% were single hospital evaluations and the number of participating staff ranged from 17 to 704. Uptake data were limited and toolkit uptake was highly variable. Studies generally indicated high satisfaction with toolkits, but the perceived usefulness of individual tools varied. Across studies, 57% reported on adherence to clinical procedures and toolkit effects were positive. Provider data were reported in 40% of studies but were primarily self-reported changes. Only 29% reported patient data and, overall, results from robust study designs are missing from the evidence base.

CONCLUSIONS

The review documents publicly available toolkits and their components. Available uptake data are limited but indicate variability. High satisfaction with toolkits can be achieved but the usefulness of individual tools may vary. The existing evidence base on the effectiveness of toolkits remains limited. While emerging evidence indicates positive effects on clinical processes, more research on toolkit value and what affects it is needed, including linking toolkits to objective provider behavior measures and patient outcomes.

TRIAL REGISTRATION

PROSPERO registration number: PROSPERO 2014: CRD42014013930 .

Do organizational cultures of Canadian medical schools promote a quality culture?

Purpose: Medical schools are expected to engage in ongoing reflection to maintain the quality of the education they deliver, that is, to cultivate a Quality Culture. Quality Culture integrates the culture of an organization with its structure and management processes. The culture of medical schools has not been previously studied. Organizational cultures can be identified using the Competing Value Questionnaire, and classified into four types, based on organizations' climate, leader style, reward systems, and strategic emphasis. Clan and Open cultures are typically positively associated with quality improvement. This study identifies the dominant organizational cultures of Canadian medical schools. Method: Sixteen of the 17 Canadian medical schools were invited to participate; one school was excluded due to ongoing accreditation activities. Faculty members of participating schools were surveyed. Results: Eleven (69%) schools participated. Nine had a dominant Hierarchical culture; two had a dominant Clan culture. Conclusions: Only two schools had a Clan culture, which might better support ongoing reflections on quality improvement. Schools leaders should examine the staff climate, leadership style, rewards system, and strategic emphasis in place at their school; these will provide clues to the existing culture and help prioritize changes required to support the implementation of a Quality Culture.

Wide-Scale Continuous Quality Improvement: A Study of Stakeholders' Use of Quality of Care Reports at Various System Levels, and Factors Mediating Use

Introduction: Increasing the use of evidence in healthcare policy and practice requires greater understanding of how stakeholders use evidence to inform policy, refine systems and change practice. Drawing on implementation theory, we have used system-focused participatory research to engage diverse stakeholders in using aggregated continuous quality improvement (CQI) data from Australian Indigenous primary health care settings to identify priority evidence-practice gaps, barriers/enablers and strategies for improvement. This article reports stakeholders' use or intended use of evidence at various levels of the system, and factors mediating use. Material and Methods: Interviews were undertaken with a purposeful sample of 30 healthcare stakeholders in different roles, organization types and settings in one Australian jurisdiction and with national participants, as part of the project's developmental evaluation. Qualitative data were analyzed to identify themes and categories relating to use of evidence. Results: Context-specific aggregated CQI data that were relatable to the diverse professional roles and practices provided an effective starting point for sharing perspectives, generating practice-based evidence and mobilizing evidence-use. Interviewees perceived the co-produced findings as applicable at different levels and useful for planning, policy development, supporting best practice and reflection, capacity strengthening and developing new research. Factors mediating use were commitment to best practice; the credibility of the evidence and its perceived relevance to work roles, contexts and decision needs; report format and language; facilitation and communication; competing work pressures and the organizational environment for change. Conclusions: This study found that primary health care stakeholders used evidence on quality of care for a variety of purposes. This could be linked to the interactive research processes used to engage stakeholders in different roles and settings in interpreting data, sharing and generating knowledge. Findings indicate that system-based participatory research using CQI data and iterative, interactive and systematic CQI-based methods can be applied at scale to support concurrent action for healthcare improvement at different system levels. Factors known to influence implementation should be addressed within the research design to optimize evidence use. Further research is needed to explore the utility of interactive dissemination for engaging healthcare stakeholders in informing policy and system change.

Continuous Quality Improvement Measured With Time-Driven Activity-Based Costing in an Outpatient Cancer Surgery Center

PURPOSE

As health care costs rise, continuous quality improvement and increased efficiency are crucial to reduce costs while providing high-quality care. Time-driven activity-based costing (TDABC) can help identify inefficiencies in processes of cancer care delivery. This study measured the process performance of Port-a-Cath placement in an outpatient cancer surgery center by using TDABC to evaluate patient care process.

METHODS

Data were collected from the Anesthesia Information Management System database and OneConnect electronic health record (EHR) for Port-a-Cath cases performed throughout four phases: preintervention (phase I), postintervention, stabilization, and pre-new EHR (phases II and III), and post-new EHR (phase IV). TDABC methods were used to map and calculate process times and costs.

RESULTS

Comparing all phases, as measured with TDABC methodology, a decrease in post-anesthesia care unit (PACU) length of stay (LOS) was identified (83 minutes v 67 minutes; P < .05). The decrease in PACU LOS correlated with increased efficiency and decreasing process costs and PACU nurse resource use by fast tracking patients for Port-a-Cath placement. Port-a-Cath placement success and the functionality of ports remained the same as patient experience improved.

CONCLUSION

TDABC can be used to evaluate processes of care delivery to patients with cancer and to quantify changes made to those processes. Patients' PACU LOS decreased on the basis of the 2013 Port-a-Cath process improvement initiative and after implementation of a new EHR, over the course of 3 years, as quantified by TDABC. TDABC use can lead to improved efficiencies in patient care delivery that are quantifiable and measurable.

A Regional ADHD Center-Based Network Project for the Diagnosis and Treatment of Children and Adolescents With ADHD

OBJECTIVE

We aimed to define the sociodemographic, clinical, and prescription profiles of the participants enrolled in the Italian Lombardy ADHD Register.

METHOD

Data on patients evaluated by the 18 regional ADHD reference centers in the 2012 to 2013 period were analyzed.

RESULTS

Seven hundred fifty-three of 1,150 (65%) suspected patients received a diagnosis of ADHD. In 24% of cases, there was a family history of ADHD. Four hundred eighty-three (64%) patients had at least one psychopathological disorder, the more common of which were learning disorders (35%). Eighty-four percent of patients received a prescription for psychoeducational interventions, 2% received only pharmacological treatment, and 14% a combination of both. Compared with patients treated with psychoeducational intervention alone, patients with drug prescriptions more commonly presented values of Clinical Global Impressions - Severity scale (CGI-S) of 5 or higher ( p < .0001).

CONCLUSION

A continuous and systematic monitoring of patterns of care is essential in promoting significant improvements in clinical practice and ensuring an efficient and homogeneous quality of care.

Implementation of continuous quality improvement in Aboriginal and Torres Strait Islander primary health care in Australia: a scoping systematic review

BACKGROUND

Continuous Quality Improvement (CQI) programs have been taken up widely by Indigenous primary health care (PHC) services in Australia and there has been national policy commitment to support this. However, international evidence shows that implementing CQI is challenging, impacts are variable and little is known about the factors that impede or enhance effectiveness. A scoping review was undertaken to explore uptake and implementation in Indigenous PHC, including barriers and enablers to embedding CQI in routine practice. We provide guidance on how research and evaluation might be intensified to support implementation.

METHODS

Searches were conducted in MEDLINE, CINAHL and the Cochrane Database of Systematic Reviews. Key websites and publications were handsearched. Studies conducted in Indigenous PHC which demonstrated some combination of CQI characteristics and assessed some aspect of implementation were included. A two stage analysis was undertaken. Stage 1 identified the breadth and focus of literature. Stage 2 investigated barriers and enablers. The Framework for Performance Assessment in PHC (2008) was used to frame the analysis. Data were extracted on the study type, approach, timeframes, CQI strategies, barriers and enablers.

RESULTS

Sixty articles were included in Stage 1 and 21 in Stage 2. Barriers to implementing CQI processes relate primarily to professional and organisational processes and operate at multiple levels (individual, team, service, health system) whereas barriers to improved care relate more directly to knowledge of best practice and team processes that facilitate appropriate care. Few studies described implementation timeframes, number of CQI cycles or improvement strategies implemented and only two applied a change theory.

CONCLUSION

Investigating barriers and enablers that modify implementation and impacts of CQI poses conceptual and methodological challenges. More complete description of CQI processes, implementation strategies, and barriers and enablers could enhance capacity for comparisons across settings and contribute to better understanding of key success factors.

Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies

OBJECTIVES

To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes.

STUDY DESIGN AND SETTING

Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels.

RESULTS

Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus.

CONCLUSION

Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.

Accreditation of Medical Education Programs: Moving From Student Outcomes to Continuous Quality Improvement Measures

Accreditation of undergraduate medical education programs aims to ensure the quality of medical education and promote quality improvement, with the ultimate goal of providing optimal patient care. Direct linkages between accreditation and education quality are, however, difficult to establish. The literature examining the impact of accreditation predominantly focuses on student outcomes, such as performances on national examinations. However, student outcomes present challenges with regard to data availability, comparability, and contamination.The true impact of accreditation may well rest in its ability to promote continuous quality improvement (CQI) within medical education programs. The conceptual model grounding this paper suggests accreditation leads medical schools to commit resources to and engage in self-assessment activities that represent best practices of CQI, leading to the development within schools of a culture of CQI. In line with this model, measures of the impact of accreditation on medical schools need to include CQI-related markers. The CQI orientation of organizations can be measured using validated instruments from the business and management fields. Repeated determinations of medical schools' CQI orientation at various points throughout their accreditation cycles could provide additional evidence of the impact of accreditation on medical education. Strong CQI orientation should lead to high-quality medical education and would serve as a proxy marker for the quality of graduates and possibly for the quality of care they provide.It is time to move away from a focus on student outcomes as measures of the impact of accreditation and embrace additional markers, such as indicators of organizational CQI orientation.

The impact of accreditation on medical schools' processes

OBJECTIVES

Increased emphasis is being placed worldwide on accreditation of undergraduate medical education programmes, and costs of participation in accreditation continue to rise. The primary purposes of accreditation are to ensure the quality of medical education and to promote quality improvement. Student performance data as indicators of the impact of accreditation have important limitations. The purpose of this study was to evaluate the impact of accreditation using an innovative marker: the processes implemented at medical schools as a result of accreditation. This conceptual model suggests that accreditation drives medical schools to implement and strengthen processes that support quality in medical education.

METHODS

In this qualitative study, conducted in 2015-2016, interviews and focus group discussions with deans, undergraduate medical education deans and faculty leaders at 13 of the 17 Canadian medical schools were used to elicit perspectives about processes influenced by accreditation; the method of constant comparative analysis associated with grounded theory was used to generate themes of processes. Perceived negative consequences of accreditation on medical education programmes were also explored.

RESULTS

Nine themes representing processes reported as resulting from accreditation were identified. These processes related to: (i) governance, (ii) data collection and analysis, (iii) monitoring, (iv) documentation, (v) creation and revision of policies and procedures, (vi) continuous quality improvement, (vii) faculty members' engagement, (viii) academic accountability and (ix) curriculum reforms. Themes representing negative consequences of accreditation included (i) costs, (ii) staff and faculty members' morale and feelings, (iii) school reputation and (iv) standards. The identified processes, given their nature, appear likely to be associated with improvement of quality in medical education. These results help justify the costs associated with accreditation.

CONCLUSIONS

This study uses an innovative marker, medical schools' processes, to evaluate the impact of accreditation. Results provide evidence that accreditation-related activities steer medical education programmes towards establishment of processes likely to be associated with improved quality in medical education.

Triage quality control is missing tools-a new observation technique for ED quality improvement

Objective

Correct assessment of patient urgency is critical to ensuring patient safety in emergency departments (EDs). Although significant time and effort have been devoted to developing triage systems, less attention has been paid to the development of quality control. The aim of this study is to introduce and test observation technique, which enables identifying of patient groups at risk of erroneous assessment in triage. The introduced technique is aimed to be less laborious to use than existing triage quality control methods.

Design

The study developed an observation technique for identifying patients with possible erroneous assessments in triage. Data sample for the observation technique is carried out with survey form filled in by nurse.

Setting

Hospital ED with ~74 000 patient visits annually.

Participants

Consecutive adult patients in an ED for baseline study period of 14 days (1774 patients) in 2010 and control study period of 4 days (541 patients) in 2012.

Intervention

Triage observation technique for continuous improvement of triage performance.

Main Outcome Measures

Primary measures of triage improvement were triage accuracy and nurses' ability to predict patient admissions.

Results

With the observation technique the ED staff was able to identify patient groups at risk for erroneous triage. Under-triage related mostly to patients with chest pain, shortness of breath, collapse, stomach pain and infections. Instead injures and muscular skeletal symptoms were seldom undertriaged even though they are common.

Conclusions

EDs can control triage quality with simple observation technique. The usability of observation technique and triage quality improvement process were good.

Perspectives: answering questions about quality improvement: suggestions for investigators

'Does it work?' is not the only question that practical improvers have of those investigating of quality improvements. They also want to know, 'Will it work here? What conditions do we need to implement and sustain it? Can we adapt it? How much will it cost and save? Is there enough evidence to spread it?'This perspectives article describes methods that investigators can use to answer these questions about improvement changes and improvement methods. It suggests that one reason why research is underused by improvers is because there is little research that answers these questions that would enable improvers to decide whether or how to implement an improvement in their local setting. It shows improvers that answers are possible and where improvers might find research and reports which answer these questions. It is based on reviews of research and reports about methods for producing valid and actionable knowledge to answer these questions. It describes a new 'quality improvement investigation movement' which is uniting applied researchers and improvers to use innovative methods to answer these questions. These investigators recognize the strengths of the randomized controlled trail method, and how easy it is to draw the wrong conclusions from data generated using lower cost and more timely methods. It emphasizes how investigators can choose a method suited to each question, describe the limitations of the method and communicate to improvers the degree of certainty of their answers to the questions.

Key Characteristics of Rehabilitation Quality Improvement Publications: Scoping Review From 2010 to 2016

OBJECTIVE

To characterize the peer-reviewed quality improvement (QI) literature in rehabilitation.

DATA SOURCES

Five electronic databases were searched for English-language articles from 2010 to 2016. Keywords for QI and safety management were searched for in combination with keywords for rehabilitation content and journals. Secondary searches (eg, references-list scanning) were also performed.

STUDY SELECTION

Two reviewers independently selected articles using working definitions of rehabilitation and QI study types; of 1016 references, 112 full texts were assessed for eligibility.

DATA EXTRACTION

Reported study characteristics including study focus, study setting, use of inferential statistics, stated limitations, and use of improvement cycles and theoretical models were extracted by 1 reviewer, with a second reviewer consulted whenever inferences or interpretation were involved.

DATA SYNTHESIS

Fifty-nine empirical rehabilitation QI studies were found: 43 reporting on local QI activities, 7 reporting on QI effectiveness research, 8 reporting on QI facilitators or barriers, and 1 systematic review of a specific topic. The number of publications had significant yearly growth between 2010 and 2016 (P=.03). Among the 43 reports on local QI activities, 23.3% did not explicitly report any study limitations; 39.5% did not used inferential statistics to measure the QI impact; 95.3% did not cite/mention the appropriate reporting guidelines; only 18.6% reported multiple QI cycles; just over 50% reported using a model to guide the QI activity; and only 7% reported the use of a particular theoretical model. Study sites and focuses were diverse; however, nearly a third (30.2%) examined early mobilization in intensive care units.

CONCLUSIONS

The number of empirical, peer-reviewed rehabilitation QI publications is growing but remains a tiny fraction of rehabilitation research publications. Rehabilitation QI studies could be strengthened by greater use of extant models and theory to guide the QI work, consistent reporting of study limitations, and use of inferential statistics.

A New Framework for Addressing Adverse Childhood and Community Experiences: The Building Community Resilience Model

OBJECTIVE

We propose a transformative approach to foster collaboration across child health, public health, and community-based agencies to address the root causes of toxic stress and childhood adversity and to build community resilience.

METHODS

Physicians, members of social service agencies, and experts in toxic stress and adverse childhood experiences (ACEs) were interviewed to inform development of the Building Community Resilience (BCR) model. Through a series of key informant interviews and focus groups, we sought to understand the role of BCR for child health systems and their partners to reduce toxic stress and build community resilience to improve child health outcomes.

RESULTS

Key informants indicated the intentional approach to ACEs and toxic stress through continuous quality improvement (data-driven decisions and program development, partners testing and adapting to changes to their needs, and iterative development and testing) which provides a mechanism by which social determinants or a population health approach could be introduced to physicians and community partners as part of a larger effort to build community resilience. Structured interviews also reveal a need for a framework that provides guidance, structure, and support for child health systems and community partners to develop collective goals, shared work plans, and a means for data-sharing to reinforce the components that will contribute to community resilience.

CONCLUSIONS

Key informant interviews and focus group dialogues revealed a deep understanding of the factors related to toxic stress and ACEs. Respondents endorsed the BCR approach as a means to explore capacity issues, reduce fragmented health care delivery, and facilitate integrated systems across partners in efforts to build community resilience. Current financing models are seen as a potential barrier, because they often do not support restructured roles, partnership development, and the work to sustain upstream efforts to address toxic stress and community resilience.

Building capacity for continuous quality improvement (CQI): A pilot study

BACKGROUND AND OBJECTIVE

Little is known about the feasibility, effectiveness, and sustainability of CQI approaches in substance use disorder treatment settings.

METHODS

In the initial phase of this study, eight programs were randomly assigned to receive a CQI intervention or to a waitlist control condition to obtain preliminary information about potential effectiveness. In the second phase, the initially assigned control programs received the CQI intervention to gain additional information about intervention feasibility while sustainability was explored among the initially assigned intervention programs.

RESULTS AND CONCLUSIONS

Although CQI was feasible and sustainable, demonstrating its effectiveness using administrative data was challenging suggesting the need to better align performance measurement systems with CQI efforts. Further, although the majority of staff were enthusiastic about utilizing this approach and reported provider and patient benefits, many noted that dedicated time was needed in order to implement and sustain it.

Ethics and Science in the Participatory Era: A Vignette-Based Delphi Study

Participatory researchers rely on relationship-based ethical considerations to address ethical challenges, but little is known about how these considerations shape day-to-day decision-making. We presented vignettes describing ethical dilemmas to health researchers with varying degrees of experience in participatory research in an online modified-Delphi panel. Panelists rated potential solutions to dilemmas and provided rationales for choosing solutions in text boxes and discussion boards. Panelists agreed on a preferred solution for three of five vignettes. Panelists indicated that ethical research decision-making should be inclusive of diverse perspectives. Panelists prioritized scientific goals above participatory ones and viewed tenets of scientific validity as value-free. Panelists indicated that participatory research ethics would be difficult to codify, but that vignette-based learning could support didactic objectives.

Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics

OBJECTIVE

Little evidence exists about effective and scalable methods for meaningful stakeholder engagement in research. We explored patient/caregiver experiences with a high-tech online engagement approach for patient-centered research prioritization, compared their experiences with those of professional stakeholders, and identified factors associated with favorable participant experiences.

METHODS

We conducted 8 online modified-Delphi (OMD) panels. Panelists participated in 2 rating rounds with a statistical feedback/online discussion round in between. Panels focused on weight management/obesity, heart failure, and Kawasaki disease. We recruited a convenience sample of adults with any of the 3 conditions (or parents/guardians of Kawasaki disease patients), clinicians, and researchers. Measures included self-reported willingness to use OMD again, the panelists' study participation and online discussion experiences, the system's perceived ease of use, and active engagement metrics.

RESULTS

Out of 349 panelists, 292 (84%) completed the study. Of those, 46% were patients, 36% were clinicians, and 19% were researchers. In multivariate models, patients were not significantly more actively engaged (Odds ratio (OR) = 1.69, 95% confidence interval (CI), 0.94-3.05) but had more favorable study participation (β = 0.49; P  ≤ .05) and online discussion (β = 0.18; P  ≤ .05) experiences and were more willing to use OMD again (β = 0.36; P  ≤ .05), compared to professional stakeholders. Positive perceptions of the OMD system's ease of use (β = 0.16; P  ≤ .05) and favorable study participation (β = 0.26; P  ≤ .05) and online discussion (β = 0.57; P  ≤ .05) experiences were also associated with increased willingness to use OMD in the future. Active engagement was not associated with online experience indices or willingness to use OMD again.

CONCLUSION

Online approaches to engaging large numbers of stakeholders are a promising and efficient adjunct to in-person meetings.

Engaging Patients and Caregivers Managing Rare Diseases to Improve the Methods of Clinical Guideline Development: A Research Protocol

BACKGROUND

Clinical guidelines provide systematically developed recommendations for deciding on appropriate health care options for specific conditions and clinical circumstances. Up until recently, patients and caregivers have rarely been included in the process of developing care guidelines.

OBJECTIVE

This project will develop and test a new online method for including patients and their caregivers in this process using Duchenne muscular dystrophy (DMD) care guidelines as an example. The new method will mirror and complement the RAND/UCLA Appropriateness Method (RAM)-the gold standard approach for conducting clinical expert panels that uses a modified Delphi format. RAM is often used in clinical guideline development to determine care appropriateness and necessity in situations where existing clinical evidence is uncertain, weak, or unavailable.

METHODS

To develop the new method for engaging patients and their caregivers in guideline development, we will first conduct interviews with experts on RAM, guideline development, patient engagement, and patient-centeredness and engage with Duchenne patients and caregivers to identify how RAM should be modified for the purposes of patient engagement and what rating criteria should patients and caregivers use to provide their input during the process of guideline development. Once the new method is piloted, we will test it by conducting two concurrently run patient/caregiver panels that will rate patient-centeredness of a subset of DMD care management recommendations already deemed clinically appropriate and necessary. The ExpertLens™ system-a previously evaluated online modified Delphi system that combines two rounds of rating with a round of feedback and moderated online discussions-will be used to conduct these panels. In addition to developing and testing the new engagement method, we will work with the members of our project's Advisory Board to generate a list of best practices for enhancing the level of patient and caregiver involvement in the guideline development process. We will solicit input on these best practice from Duchenne patients, caregivers, and clinicians by conducting a series of round-table discussions and making a presentation at an annual conference on Duchenne.

RESULTS

The study protocol was reviewed by RAND's Human Subjects Protection Committee, which determined it to be exempt from review. Interviews with RAM experts have been completed. The projected study completion date is May 2020.

CONCLUSIONS

We expect that the new method will make it easier to engage large numbers of patients and caregivers in the process of guideline development in a rigorous and culturally appropriate manner that is consistent with the way clinicians participate in guideline development. Moreover, this project will develop best practices that could help involve patients and caregivers in the clinical guideline development process in other clinical areas, thereby facilitating the work of guideline developers.

Acceptability of an online modified Delphi panel approach for developing health services performance measures: results from 3 panels on arthritis research

RATIONALE, AIMS, AND OBJECTIVES

Online modified Delphi (OMD) panel approaches can be used to engage large and diverse groups of clinical experts and stakeholders in developing health services performance measures. Such approaches are increasing in popularity among health researchers. However, information about their acceptability to participating experts and stakeholders is lacking but important to determine before recommending widespread use of online approaches. Therefore, the objective of this paper is to explore acceptability of the OMD panel approach from the participants' perspective.

METHOD

We use data from participants in three OMD panels designed to develop performance measures for use in arthritis research and quality improvement efforts. At the end of each online panel, we surveyed clinical experts and stakeholders who shared their experiences with the OMD process by answering 13 close-ended questions using 7-point Likert-type scales. A mean of 5 or higher on a given question was treated as an indication of acceptability.

RESULTS

Ninety-eight clinical experts and stakeholders (92% participation rate) answered survey questions about the online process. They considered the OMD panel approach to be acceptable, particularly the ease of using the online system (mean = 5.3, standard deviation = 1.3) and the understanding gained from online discussions (mean = 5.2, standard deviation = 1.0). Participants also felt that participation in the Delphi study was interesting (mean = 5.6, standard deviation =1.1).

CONCLUSION

These findings illustrate likely acceptability and a potential for a more widespread use of OMD panel approaches by stakeholders in developing health services performance measures.

Screening Tool of Older Person's Prescriptions/Screening Tools to Alert Doctors to Right Treatment Medication Criteria Modified for U.S. Nursing Home Setting

OBJECTIVES

To develop a set of prescribing indicators measurable with available data from electronic nursing home (NH) databases by adapting the European-based 2014 Screening Tool of Older Person's Prescriptions (STOPP) and Screening Tools to Alert Doctors to Right Treatment (START) criteria of potentially inappropriate and underused medications for the U.S.

SETTING

DESIGN

A two-stage expert panel process. In the first stage, the investigator team reviewed 114 criteria for compatibility and measurability. In the second stage, an online modified e-Delphi (OMD) panel was convened to rate the validity of criteria, and two webinars were held to identify criteria with highest relevance to U.S. NHs.

PARTICIPANTS

Seventeen experts with recognized reputations in NH care participated in the e-Delphi panel and 12 in the webinar.

MEASUREMENTS

Compatibility and measurability were assessed by comparing criteria with U.S. terminology and setting standards and data elements in NH databases. Validity was rated using a 9-point Likert-type scale (1 = not valid at all, 9 = highly valid). Mean, median, interpercentile ranges, and agreement were determined for each criterion score. Relevance was determined by ranking the mean panel ratings on criteria that reached agreement; the webinar participants reviewed and approved half of the criteria with the highest mean values.

RESULTS

Fifty-three STOPP/START criteria were deemed to be compatible with the U.S. NH setting and measurable using data from electronic NH databases. E-Delphi panelists rated 48 criteria as valid for U.S. NHs. Twenty-four criteria were deemed to be most relevant, consisting of 22 measures of potentially inappropriate medications and two measures of underused medications.

CONCLUSION

This study created the first explicit criteria for assessing the quality of prescribing in U.S. NHs.

On using ethical principles of community-engaged research in translational science

The transfer of new discoveries into both clinical practice and the wider community calls for reliance on interdisciplinary translational teams that include researchers with different areas of expertise, representatives of health care systems and community organizations, and patients. Engaging new stakeholders in research, however, calls for a reconsideration or expansion of the meaning of ethics in translational research. We explored expert opinion on the applicability of ethical principles commonly practiced in community-engaged research (CEnR) to translational research. To do so, we conducted 2 online, modified-Delphi panels with 63 expert stakeholders who iteratively rated and discussed 9 ethical principles commonly used in CEnR in terms of their importance and feasibility for use in translational research. The RAND/UCLA appropriateness method was used to analyze the data and determine agreement and disagreement among participating experts. Both panels agreed that ethical translational research should be "grounded in trust." Although the academic panel endorsed "culturally appropriate" and "forthcoming with community about study risks and benefits," the mixed academic-community panel endorsed "scientifically valid" and "ready to involve community in interpretation and dissemination" as important and feasible principles of ethical translational research. These findings suggest that in addition to protecting human subjects, contemporary translational science models need to account for the interests of, and owe ethical obligations to, members of the investigative team and the community at large.

Patient engagement in the process of planning and designing outpatient care improvements at the Veterans Administration Health-care System: findings from an online expert panel

CONTEXT

There is a strong interest in the Veterans Administration (VA) Health-care System in promoting patient engagement to improve patient care.

METHODS

We solicited expert opinion using an online expert panel system with a modified Delphi structure called ExpertLens^™ . Experts reviewed, rated and discussed eight scenarios, representing four patient engagement roles in designing and improving VA outpatient care (consultant, implementation advisor, equal stakeholder and lead stakeholder) and two VA levels (local and regional). Rating criteria included desirability, feasibility, patient ability, physician/staff acceptance and impact on patient-centredness and care quality. Data were analysed using the RAND/UCLA Appropriateness Method for determining consensus.

FINDINGS

Experts rated consulting with patients at the local level as the most desirable and feasible patient engagement approach. Engagement at the local level was considered more desirable than engagement at the regional level. Being an equal stakeholder at the local level received the highest ratings on the patient-centredness and health-care quality criteria.

CONCLUSIONS

Our findings illustrate expert opinion about different approaches to patient engagement and highlight the benefits and challenges posed by each. Although experts rated local consultations with patients on an as-needed basis as most desirable and feasible, they rated being an equal stakeholder at the local level as having the highest potential impact on patient-centredness and care quality. This result highlights a perceived discrepancy between what is most desirable and what is potentially most effective, but suggests that routine local engagement of patients as equal stakeholders may be a desirable first step for promoting high-quality, patient-centred care.

Development of System-level Performance Measures for Evaluation of Models of Care for Inflammatory Arthritis in Canada

Objective.

To develop system-level performance measures for evaluating the care of patients with inflammatory arthritis (IA), including rheumatoid arthritis (RA), psoriatic arthritis, ankylosing spondylitis, and juvenile idiopathic arthritis.

Methods.

This study involved several methodological phases. Over multiple rounds, various participants were asked to help define a set of candidate measurement themes. A systematic search was conducted of existing guidelines and measures. A set of 6 performance measures was defined and presented to 50 people, including patients with IA, rheumatologists, allied health professionals, and researchers using a 3-round, online, modified Delphi process. Participants rated the validity, feasibility, relevance, and likelihood of use of the measures. Measures with median ratings ≥ 7 for validity and relevance were included in the final set.

Results.

Six performance measures were developed evaluating the following aspects of care, with each measure being applied separately for each type of IA except where specified: waiting times for rheumatology consultation for patients with new onset IA, percentage of patients with IA seen by a rheumatologist, percentage of patients with IA seen in yearly followup by a rheumatologist, percentage of patients with RA treated with a disease-modifying antirheumatic drug (DMARD), time to DMARD therapy in RA, and number of rheumatologists per capita.

Conclusion.

The first set of system-level performance measures for IA care in Canada has been developed with broad input. The measures focus on timely access to care and initiation of appropriate treatment for patients with IA, and are likely to be of interest to other arthritis care systems internationally.

Furthering the quality agenda in Aboriginal community controlled health services: understanding the relationship between accreditation, continuous quality improvement and national key performance indicator reporting

A rapidly expanding interest in quality in the Aboriginal-community-controlled health sector has led to widespread uptake of accreditation using more than one set of standards, a proliferation of continuous quality improvement programs and the introduction of key performance indicators. As yet, there has been no overarching logic that shows how they relate to each other, with consequent confusion within and outside the sector. We map the three approaches to the Framework for Performance Assessment in Primary Health Care, demonstrating their key differences and complementarity. There needs to be greater attention in both policy and practice to the purposes and alignment of the three approaches if they are to embed a system-wide focus that supports quality improvement at the service level.

Community-acquired pneumonia (CAP) hospitalizations and deaths: is there a role for quality improvement through inter-hospital comparisons?

OBJECTIVE

To assess between-hospital variations in standardized in-hospital mortality ratios of community-acquired pneumonia (CAP), and identify possible leads for quality improvement.

DESIGN

We used an administrative database to estimate standardized in-hospital mortality ratios for 111 Belgian hospitals, by carrying out a set of hierarchical logistic regression models, intended to disentangle therapeutic attitudes and biases. To facilitate the detection of false-negative/positive results, we added an inconclusive zone to the funnel plots, derived from the results of the study. Data quality was validated by comparison with (i) alternative data from the largest Belgian Sickness Fund, (ii) published German hospital data and (iii) the results of an on-site audit.

SETTING

All Belgian hospital discharge records from 2004 to 2007.

STUDY PARTICIPANTS

A total of 111 776 adult patients were admitted for CAP.

MAIN OUTCOME MEASURE

Risk-adjusted standardized in-hospital mortality ratios.

RESULTS

Out of the 111 hospitals, we identified five and six outlying hospitals, with standardized mortality ratios of CAP consistently on the extremes of the distribution, as providing possibly better or worse care, respectively, and 18 other hospitals as having possible quality weaknesses/strengths. At the individuals' level of the analysis, adjusted odds ratios showed the paramount importance of old age, comorbidity and mechanical ventilation. The data compared well with the different validation sources.

CONCLUSIONS

Despite the limitations inherent to administrative data, it seemed possible to establish inter-hospital differences in standardized in-hospital mortality ratios of CAP and to identify leads for quality improvement. Monitoring is needed to assess progress in quality.

Effect of Educational Outreach Timing and Duration on Facility Performance for Infectious Disease Care in Uganda: A Trial with Pre-Post and Cluster Randomized Controlled Components

BACKGROUND

Classroom-based learning is often insufficient to ensure high quality care and application of health care guidelines. Educational outreach is garnering attention as a supplemental method to enhance health care worker capacity, yet there is little information about the timing and duration required to improve facility performance. We sought to evaluate the effects of an infectious disease training program followed by either immediate or delayed on-site support (OSS), an educational outreach approach, on nine facility performance indicators for emergency triage, assessment, and treatment; malaria; and pneumonia. We also compared the effects of nine monthly OSS visits to extended OSS, with three additional visits over six months.

METHODS

This study was conducted at 36 health facilities in Uganda, covering 1,275,960 outpatient visits over 23 months. From April 2010 to December 2010, 36 sites received infectious disease training; 18 randomly selected sites in arm A received nine monthly OSS visits (immediate OSS) and 18 sites in arm B did not. From March 2011 to September 2011, arm A sites received three additional visits every two months (extended OSS), while the arm B sites received eight monthly OSS visits (delayed OSS). We compared the combined effect of training and delayed OSS to training followed by immediate OSS to determine the effect of delaying OSS implementation by nine months. We also compared facility performance in arm A during the extended OSS to immediate OSS to examine the effect of additional, less frequent OSS.

RESULTS

Delayed OSS, when combined with training, was associated with significant pre/post improvements in four indicators: outpatients triaged (44% vs. 87%, aRR = 1.54, 99% CI = 1.11, 2.15); emergency and priority patients admitted, detained, or referred (16% vs. 31%, aRR = 1.74, 99% CI = 1.10, 2.75); patients with a negative malaria test result prescribed an antimalarial (53% vs. 34%, aRR = 0.67, 99% CI = 0.55, 0.82); and pneumonia suspects assessed for pneumonia (6% vs. 27%, aRR = 2.97, 99% CI = 1.44, 6.17). Differences between the delayed OSS and immediate OSS arms were not statistically significant for any of the nine indicators (all adjusted relative RR (aRRR) between 0.76-1.44, all p>0.06). Extended OSS was associated with significant improvement in two indicators (outpatients triaged: aRR = 1.09, 99% CI = 1.01; emergency and priority patients admitted, detained, or referred: aRR = 1.22, 99% CI = 1.01, 1.38) and decline in one (pneumonia suspects assessed for pneumonia: aRR: 0.93; 99% CI = 0.88, 0.98).

CONCLUSIONS

Educational outreach held up to nine months after training had similar effects on facility performance as educational outreach started within one month post-training. Six months of bi-monthly educational outreach maintained facility performance gains, but incremental improvements were heterogeneous.

Development of the Quality Improvement Minimum Quality Criteria Set (QI-MQCS): a tool for critical appraisal of quality improvement intervention publications

Objective

Valid, reliable critical appraisal tools advance quality improvement (QI) intervention impacts by helping stakeholders identify higher quality studies. QI approaches are diverse and differ from clinical interventions. Widely used critical appraisal instruments do not take unique QI features into account and existing QI tools (eg, Standards for QI Reporting Excellence) are intended for publication guidance rather than critical appraisal. This study developed and psychometrically tested a critical appraisal instrument, the QI Minimum Quality Criteria Set (QI-MQCS) for assessing QI-specific features of QI publications.

Methods

Approaches to developing the tool and ensuring validity included a literature review, in-person and online survey expert panel input, and application to empirical examples. We investigated psychometric properties in a set of diverse QI publications (N=54) by analysing reliability measures and item endorsement rates and explored sources of disagreement between reviewers.

Results

The QI-MQCS includes 16 content domains to evaluate QI intervention publications: Organisational Motivation, Intervention Rationale, Intervention Description, Organisational Characteristics, Implementation, Study Design, Comparator Description, Data Sources, Timing, Adherence/Fidelity, Health Outcomes, Organisational Readiness, Penetration/Reach, Sustainability, Spread and Limitations. Median inter-rater agreement for QI-MQCS items was κ 0.57 (83% agreement). Item statistics indicated sufficient ability to differentiate between publications (median quality criteria met 67%). Internal consistency measures indicated coherence without excessive conceptual overlap (absolute mean interitem correlation=0.19). The critical appraisal instrument is accompanied by a user manual detailing What to consider, Where to look and How to rate.

Conclusions

We developed a ready-to-use, valid and reliable critical appraisal instrument applicable to healthcare QI intervention publications, but recognise scope for continuing refinement.

Development of Cardiovascular Quality Indicators for Rheumatoid Arthritis: Results from an International Expert Panel Using a Novel Online Process

OBJECTIVE

Patients with rheumatoid arthritis (RA) have a high risk of premature cardiovascular disease (CVD). We developed CVD quality indicators (QI) for screening and use in rheumatology clinics.

METHODS

A systematic review was conducted of the literature on CVD risk reduction in RA and the general population. Based on the best practices identified from this review, a draft set of 12 candidate QI were presented to a Canadian panel of rheumatologists and cardiologists (n = 6) from 3 academic centers to achieve consensus on the QI specifications. The resulting 11 QI were then evaluated by an online modified-Delphi panel of multidisciplinary health professionals and patients (n = 43) to determine their relevance, validity, and feasibility in 3 rounds of online voting and threaded discussion using a modified RAND/University of California, Los Angeles Appropriateness Methodology.

RESULTS

Response rates for the online panel were 86%. All 11 QI were rated as highly relevant, valid, and feasible (median rating ≥ 7 on a 1-9 scale), with no significant disagreement. The final QI set addresses the following themes: communication to primary care about increased CV risk in RA; CV risk assessment; defining smoking status and providing cessation counseling; screening and addressing hypertension, dyslipidemia, and diabetes; exercise recommendations; body mass index screening and lifestyle counseling; minimizing corticosteroid use; and communicating to patients at high risk of CVD about the risks/benefits of nonsteroidal antiinflammatory drugs.

CONCLUSION

Eleven QI for CVD care in patients with RA have been developed and are rated as highly relevant, valid, and feasible by an international multidisciplinary panel.