Performance measurement and improvement frameworks in health, education and social services systems: a systematic review

Adapting the Donabedian model in undergraduate nursing education: a modified Delphi study

BACKGROUND

Donabedian conceptual and multidimensional framework has been used in several studies in an educational context. In this study, we aimed to adapt the Donabedian three-component model (structure, process, and outcome) in undergraduate nursing education. This conceptual framework provides a comprehensive image of nursing education institutions and can help to evaluate institutions by measuring different aspects of their performance. A comprehensive understanding of the various elements of an educational institution helps to develop a complete, appropriate relevant set of performance indicators.

METHODS

This was a modified Delphi study. It had three rounds. The expert panel consisted of nursing faculty members and nursing Ph.D. students. In the first round, a questionnaire was designed based on interviews, focus groups, and a literature review. Experts rated their agreement with each element on a 5-point Likert scale in rounds two and three. The consensus level was set as 75%. The stability between rounds was also determined by calculating kappa coefficients. One Sample T-Test was also calculated for new items in round three.

RESULTS

All 55 items of the questionnaire were confirmed in the second round based on the consensus percentage of 75. Five new items were added to the third round based on comments in round two. Eventually, all elements except one were confirmed according to the consensus level, kappa values, means, and One-Sample T-Test in round three. The structure's key elements include staff (academic and non-academic); equipment; guidelines; resources and facilities; and students' demographics and characteristics. Process key elements include communication; education; evaluation; cooperation; and consultation. Outcome key elements include knowledge development; nursing image; alumni's outcome; students' outcome; related medical centers' performance; accreditation and evaluation results; and satisfaction.

CONCLUSIONS

Different elements of a nursing education institution at the bachelor's level were determined. The results of this study can help related bodies to develop and implement a comprehensive and systematic evaluation. These results can also be a basis for making this model useful in other nursing courses or education in other fields.

A Conceptual Framework for Evaluating National Organ Donation and Transplantation Programs

Conceptual frameworks are valuable resources that can be used to guide the planning, evaluation, and development of healthcare services. However, there are currently no comprehensive frameworks focused on organ donation and transplantation that identify the critical factors underlying a successful national program. To address this knowledge gap, we developed a conceptual framework that takes into account all major domains of influence, including political and societal aspects as well as clinical implementation. The framework was initially constructed based on a targeted review of the relevant medical literature. Feedback provided by a panel of international experts was incorporated into the framework via an iterative process. The final framework features 16 essential domains that are critical for initiating and maintaining a successful program and improving the health of patients with organ failure. Of particular note, these domains are subject to three overarching health system principles: responsiveness, efficiency, and equity. This framework represents a first attempt to develop a whole-system view of the various factors that contribute to the success of a national program. These findings provide a useful tool that can be adapted to any jurisdiction and used to plan, evaluate, and improve organ donation and transplantation programs.

Gains and losses within the homeless service, supportive housing, and harm reduction sectors during the COVID-19 pandemic: A qualitative study of what matters to the workforce

The COVID-19 pandemic has had deleterious effects on individuals experiencing homelessness; yet, less is known about how this global health crisis is impacting service providers that support the homeless population. This qualitative study examined the perceived impacts of the COVID-19 pandemic on the lives and work experiences of service providers in the homeless service, supportive housing, and harm reduction sectors in Canada. Further analyses were conducted to identify the occupational values that were represented in the work-related changes experienced by providers. A stratified purposive sample of 40 participants (30 direct service providers and 10 providers in leadership roles) were drawn from a pan-Canadian study of the mental health of service providers working with individuals experiencing homelessness. Reflexive thematic analysis was used to identify five themes of the work-related changes experienced by service providers during the pandemic: [1] "Everything was changing every day": Work role and responsibility instability; [2] "How on Earth do we do our job?": Challenges to working relationships with service users; [3] "It used to be a social environment": Transitions to impersonal and isolating workspaces; [4] "It all comes down the chute": Lack of organisational support and hierarchical conflict; and [5] "We've been supported as well as we could have": Positive organisational support and communication. The findings underscored how many of the occupational changes during the pandemic did not align with service providers' occupational values for collaboration, control, effective and safe service provision, and the importance of human relationships, among other values. As pre-existing sectoral problems were exacerbated by the pandemic, recovery efforts need to address these long-standing issues in ways that are aligned with service providers' values. Future research is warranted on how organisational approaches can promote supportive workplaces for service providers and improve outcomes for individuals experiencing homelessness.

A systematic review: the dimensions to evaluate health care performance and an implication during the pandemic

BACKGROUND

The balanced scorecard (BSC) has been implemented to evaluate the performance of health care organizations (HCOs). BSC proved to be effective in improving financial performance and patient satisfaction.

AIM

This systematic review aims to identify all the perspectives, dimensions, and KPIs that are vital and most frequently used by health care managers in BSC implementations.

METHODS

This systematic review adheres to PRISMA guidelines. The PubMed, Embase, Cochrane, and Google Scholar databases and Google search engine were inspected to find all implementations of BSC at HCO. The risk of bias was assessed using the nonrandomized intervention studies (ROBINS-I) tool to evaluate the quality of observational and quasi-experimental studies and the Cochrane (RoB 2) tool for randomized controlled trials (RCTs).

RESULTS

There were 33 eligible studies, of which we identified 36 BSC implementations. The categorization and regrouping of the 797 KPIs resulted in 45 subdimensions. The reassembly of these subdimensions resulted in 13 major dimensions: financial, efficiency and effectiveness, availability and quality of supplies and services, managerial tasks, health care workers' (HCWs) scientific development error-free and safety, time, HCW-centeredness, patient-centeredness, technology, and information systems, community care and reputation, HCO building, and communication. On the other hand, this review detected that BSC design modification to include external and managerial perspectives was necessary for many BSC implementations.

CONCLUSION

This review solves the KPI categorization dilemma. It also guides researchers and health care managers in choosing dimensions for future BSC implementations and performance evaluations in general. Consequently, dimension uniformity will improve the data sharing and comparability among studies. Additionally, despite the pandemic negatively influencing many dimensions, the researchers observed a lack of comprehensive HCO performance evaluations. In the same vein, although some resulting dimensions were assessed separately during the pandemic, other dimensions still lack investigation. Last, BSC dimensions may play an essential role in tackling the COVID-19 pandemic. However, further research is required to investigate the BSC implementation effect in mitigating the pandemic consequences on HCO.

Terminology and descriptions of navigation and related practices for children with neurodisability and their families: a scoping review

BACKGROUND

Children with neurodisability (ND) represent a significant population with a demonstrated need for coordinated support. Patient navigation has a primary focus on: facilitating access to and connection amongst fragmented systems; as well as the provision of educational and emotional support. Given the distinct needs of children with ND and their families, programs built upon such core concepts could be of great benefit. The diversity of terminology encompassing navigation-related concepts and activities (e.g., care coordination, case management, family support), however, presents challenges to both practice and research. This scoping review examined the terminology and descriptions provided within published articles on navigation-type models for children with ND and their families.

METHODS

The scoping review was conducted according to the Joanna Briggs Institute methodology. A preliminary search was completed on PubMed (NCBI), MEDLINE (Ovid) and CINAHL (EBSCO) to identify initial search terms, upon which a full search strategy was developed and executed in MEDLINE (Ovid) and CINAHL (EBSCO). After screening records according to our inclusion and exclusion criteria, a full-text review of relevant articles was conducted and data extracted using a researcher-developed tool. Under close supervision by the research team, study selection was primarily performed by one author.

RESULTS

Of the 2597 papers identified, 33 were included in the final review. From the included papers, a total of 49 terms were extracted, 20 of which were unique. Across the diversity of terminology observed, articles provided detailed and rich descriptions characterized by four central domains, namely: (i) what navigation-related resources, supports and services aim to facilitate and (ii) provide; (iii) descriptions of their intended outcomes; as well as (iv) guiding principles.

CONCLUSIONS

This scoping review addresses a gap in our knowledge related to the specification of patient navigation and related supports as applied to the specific context of children with ND and their families. Given the particular needs of this population, we propose an empirically-informed integrative model that synthesizes the findings from this scoping review. We suggest that this framework can be used as a guide to the mindful characterization of how supports aiming to connect children and families to needed service are termed and described within future research and in practice.

European Society of Cardiology methodology for the development of quality indicators for the quantification of cardiovascular care and outcomes

AIMS

It is increasingly recognized that tools are required for assessing and benchmarking quality of care in order to improve it. The European Society of Cardiology (ESC) is developing a suite of quality indicators (QIs) to evaluate cardiovascular care and support the delivery of evidence-based care. This paper describes the methodology used for their development.

METHODS AND RESULTS

We propose a four-step process for the development of the ESC QIs. For a specific clinical area with a gap in care delivery, the QI development process includes: (i) the identification of key domains of care by constructing a conceptual framework of care; (ii) the construction of candidate QIs by conducting a systematic review of the literature; (iii) the selection of a final set of QIs by obtaining expert opinions using the modified Delphi method; and (iv) the undertaking of a feasibility assessment by evaluating different ways of defining the QI specifications for the proposed data collection source. For each of the four steps, key methodological areas need to be addressed to inform the implementation process and avoid misinterpretation of the measurement results.

CONCLUSION

Detailing the methodology for the ESC QIs construction enables healthcare providers to develop valid and feasible metrics to measure and improve the quality of cardiovascular care. As such, high-quality evidence may be translated into clinical practice and the 'evidence-practice' gap closed.

What makes a good quality indicator set? A systematic review of criteria

BACKGROUND

While single indicators measure a specific aspect of quality (e.g. timely support during labour), users of these indicators, such as patients, providers and policy-makers, are typically interested in some broader construct (e.g. quality of maternity care) whose measurement requires a set of indicators. However, guidance on desirable properties of indicator sets is lacking.

OBJECTIVE

Based on the premise that a set of valid indicators does not guarantee a valid set of indicators, the aim of this review is 2-fold: First, we introduce content validity as a desirable property of indicator sets and review the extent to which studies in the peer-reviewed health care quality literature address this criterion. Second, to obtain a complete inventory of criteria, we examine what additional criteria of quality indicator sets were used so far.

METHODS

We searched the databases Web of Science, Medline, Cinahl and PsycInfo from inception to May 2021 and the reference lists of included studies. English- or German-language, peer-reviewed studies concerned with desirable characteristics of quality indicator sets were included. Applying qualitative content analysis, two authors independently coded the articles using a structured coding scheme and discussed conflicting codes until consensus was reached.

RESULTS

Of 366 studies screened, 62 were included in the review. Eighty-five per cent (53/62) of studies addressed at least one of the component criteria of content validity (content coverage, proportional representation and contamination) and 15% (9/62) addressed all component criteria. Studies used various content domains to structure the targeted construct (e.g. quality dimensions, elements of the care pathway and policy priorities), providing a framework to assess content validity. The review revealed four additional substantive criteria for indicator sets: cost of measurement (21% [13/62] of the included studies), prioritization of 'essential' indicators (21% [13/62]), avoidance of redundancy (13% [8/62]) and size of the set (15% [9/62]). Additionally, four procedural criteria were identified: stakeholder involvement (69% [43/62]), using a conceptual framework (44% [27/62]), defining the purpose of measurement (26% [16/62]) and transparency of the development process (8% [5/62]).

CONCLUSION

The concept of content validity and its component criteria help assessing whether conclusions based on a set of indicators are valid conclusions about the targeted construct. To develop a valid indicator set, careful definition of the targeted construct including its (sub-)domains is paramount. Developers of quality indicators should specify the purpose of measurement and consider trade-offs with other criteria for indicator sets whose application may reduce content validity (e.g. costs of measurement) in light thereof.

Peer groups for organisational learning: Clustering with practical constraints

Peer-grouping is used in many sectors for organisational learning, policy implementation, and benchmarking. Clustering provides a statistical, data-driven method for constructing meaningful peer groups, but peer groups must be compatible with business constraints such as size and stability considerations. Additionally, statistical peer groups are constructed from many different variables, and can be difficult to understand, especially for non-statistical audiences. We developed methodology to apply business constraints to clustering solutions and allow the decision-maker to choose the balance between statistical goodness-of-fit and conformity to business constraints. Several tools were utilised to identify complex distinguishing features in peer groups, and a number of visualisations are developed to explain high-dimensional clusters for non-statistical audiences. In a case study where peer group size was required to be small (≤ 100 members), we applied constrained clustering to a noisy high-dimensional data-set over two subsequent years, ensuring that the clusters were sufficiently stable between years. Our approach not only satisfied clustering constraints on the test data, but maintained an almost monotonic negative relationship between goodness-of-fit and stability between subsequent years. We demonstrated in the context of the case study how distinguishing features between clusters can be communicated clearly to different stakeholders with substantial and limited statistical knowledge.

Access to paediatric cancer care treatment in Mexico: responding to health system challenges and opportunities

In Mexico, paediatric cancer is the leading cause of death for children aged 0-18 years. This study analyses the main challenges for paediatric cancer care from the perspective of three key health systems functions: stewardship, financing and service delivery. The study used a mixed methods approach comprised of: (1) a scoping literature review, (2) an analysis of 2008-18 expenditures on paediatric cancer by the Fund for Protection against Catastrophic Expenditures (FPGC) of Seguro Popular and (3) a nation-wide survey of the supply capacity of 59 Ministry of Health (MoH) and 39 Mexican Institute of Social Security (IMSS) hospitals engaged in paediatric cancer care. The study found that while Mexico has made substantial progress towards universal health coverage (UHC) for paediatric cancer treatment, serious gaps persist. FPGC funds for paediatric cancer increased from 2008 to 2011 to reach US$36 million and then declined to US$13.6 million in 2018, along with the number of covered cases. The distribution of health professionals and paediatric oncology infrastructure is uneven between MoH and IMSS hospitals and across Mexican regions. Both institutions share common barriers for continuous and co-ordinated health care and lack monitoring activities that cripple their capacity to apply uniform standards for high-quality cancer care. In conclusion, achieving universal and effective coverage of paediatric cancer treatment is a critical component of UHC for Mexico. This requires periodic and ongoing assessment of health system performance specific to paediatric cancer to identify gaps and propose strategies for continued investment and improvement of access to care and health outcomes for this important cause of premature mortality.

Analysing synthesis of evidence in a systematic review in health professions education: observations on struggling beyond Kirkpatrick

Background: Systematic reviews in health professions education may well under-report struggles to synthesize disparate evidence that defies standard quantitative approaches. This paper reports further process analysis in a previously reported systematic review about mobile devices on clinical placements.Objective: For a troublesome systematic review: (1) Analyse further the distribution and reliability of classifying the evidence to Maxwell quality dimensions (beyond 'Does it work?') and their overlap with Kirkpatrick K-levels. (2) Analyse how the abstracts represented those dimensions of the evidence-base. (3) Reflect on difficulties in synthesis and merits of Maxwell dimensions.Design: Following integrative synthesis of 45 K2-K4 primary studies (by combined content-thematic analysis in the pragmatism paradigm): (1) Hierarchical cluster analysis explored overlap between Maxwell dimensions and K-levels. Independent and consensus-coding to Maxwell dimensions compared (using: percentages; kappa; McNemar hypothesis-testing) pre- vs post-discussion and (2) article abstract vs main body. (3) Narrative summary captured process difficulties and merits.Results: (1) The largest cluster (five-cluster dendrogram) was acceptability-accessibility-K1-appropriateness-K3, with K1 and K4 widely separated. For article main bodies, independent coding agreed most for appropriateness (good; adjusted kappa = 0.78). Evidence increased significantly pre-post-discussion about acceptability (p = 0.008; 31/45→39/45), accessibility, and equity-ethics-professionalism. (2) Abstracts suggested efficiency significantly less than main bodies evidenced: 31.1% vs 44.4%, p = 0.031. 3) Challenges and merits emerged for before, during, and after the review.Conclusions: There should be more systematic reporting of process analysis about difficulties synthesizing suboptimal evidence-bases. In this example, Maxwell dimensions were a useful framework beyond K-levels for classifying and synthesizing the evidence-base.

The north is not all the same: comparing health system performance in 18 northern regions of Canada

We investigated the availability of health system performance indicator data in Canada’s 18 northern regions and the feasibility of using the performance framework developed by the Canadian Institute for Health Information [CIHI]. We examined the variation in 24 indicators across regions and factors that might explain such variation. The 18 regions vary in population size and various measures of socioeconomic status, health-care delivery, and health status. The worst performing health systems generally include Nunavut and the northern regions of Québec, Manitoba and Saskatchewan where indigenous people constitute the overwhelming majority of the population, ranging from 70% to 90%, and where they also fare worst in terms of adverse social determinants. All northern regions perform worse than Canada nationally in hospitalisations for ambulatory care sensitive conditions and potentially avoidable mortality. Population size, socioeconomic status, degree of urbanisation and proportion of Aboriginal people in the population are all associated with performance. The North is far from homogenous. Inter-regional variation demands further investigation. The more intermediate pathways, especially between health system inputs, outputs and outcomes, are largely unexplored. Improvement of health system performance for northern and remote regions will require the engagement of indigenous leadership, communities and patient representatives.

Development of Quality Management Capacity in Child-Serving Nonprofit Agencies

Quality or performance management capabilities allow agencies to identify effective practices in routine care, implement new practices, and learn to adapt practices as contexts change. Within child-serving human service systems there is not a dominant model of quality management capabilities and how they are deployed. Quality management capabilities and their development were explored at nine different child serving agencies. Agency respondents described four emergent core quality management capabilities: generating shared goals, managing information, routinizing problem-solving, and propagating a culture of quality. None of the nine agencies we studied excelled at all four. Each capability is described and implications for research, policy and practice are discussed.

Effective quality systems: implementation in Australian public hospitals

PURPOSE

The purpose of this paper is to review the implementation of seven components of quality systems (QSs) linked with quality improvement in a sample of Australian hospitals.

DESIGN/METHODOLOGY/APPROACH

The authors completed a systematic review to identify QS components associated with measureable quality improvement. Using mixed methods, the authors then reviewed the current state of these QS components in a sample of eight Australian hospitals.

FINDINGS

The literature review identified seven essential QS components. Both the self-evaluation and focus group data suggested that none of the hospitals had all of these seven components in place, and that there were some implementation issues with those components that were in use. Although board and senior executives could point to a large number of quality and safety documents that they felt were supporting a vision and framework for safe, high-quality care, middle managers and clinical staff described the QSs as compliance driven and largely irrelevant to their daily pursuit of safe, high-quality care. The authors also found little specific training in quality improvement for staff, lack of useful data for clinicians on the quality of care they provide and confusion about how organisational QSs work.

PRACTICAL IMPLICATIONS

This study provides a clearer picture of why QSs are not yet achieving the results that boards and executives want to achieve, and that patients require.

ORIGINALITY/VALUE

This is the first study to explore the implementation of QSs in hospitals in-depth from the perspective of hospital staff, linking the findings to the implementation of QS component identified in the literature.

Identifying health policy and systems research priorities on multisectoral collaboration for health in low-income and middle-income countries

Introduction

While efforts to achieve Universal Health Coverage (UHC) and the Sustainable Development Goals (SDGs) have reinvigorated interest in multisectoral collaborations (MSCs) among the global health and development community, there remains a plethora of questions about how best to conceptualise, plan, implement, evaluate and sustain MSCs. The objective of this paper is to present research priorities on MSC for health from researchers and policymakers around the globe, with an emphasis on low-income and middle-income countries.

Methods

The authors identified 30 priority research questions from two sources: (1) 38 review articles on MSC for health, and (2) interviews and focus groups with a total of 81 policymakers, including government officials (largely from ministries of health and state/provincial departments of health, but also offices of planning, public service, social development, the prime minister and others), large multilateral or bilateral organisations, and non-governmental organisations. In a third phase, questions were refined and ranked by a diverse group of researchers from around the globe using an online voting platform.

Results

The top-ranked questions focused predominantly on pragmatic questions, such as how best to structure, implement and sustain MSCs, as well as how to build stakeholder capacity and community partnerships. Despite substantial variation between review articles, policymakers’ reflections and online ranking by researchers, two topics emerged as research priorities for all three: (1) leadership, partnership and governance structures for MSCs; and (2) MSC implementation strategies and mechanisms. The review articles underscored the need for more guidance on appropriate study designs and methods for investigating MSCs, which may be a prerequisite for other identified research priorities.

Conclusion

These findings could inform efforts within and beyond the health sector to better align research objectives and funding with the evidence needs of policymakers grappling with questions about how best to leverage MSCs to achieve UHC and the SDGs.

Using concept mapping to develop a human rights based indicator framework to assess country efforts to strengthen rehabilitation provision and policy: the Rehabilitation System Diagnosis and Dialogue framework (RESYST)

BACKGROUND

Rehabilitation is crucial for the realization of the right to health and a proper concern of global health. Yet, reliable information to guide rehabilitation service planning is unavailable in many countries in part due to the lack of appropriate indicators. To ensure universal health coverage and meet the central imperative of "leaving no one behind" countries must be able to assess key aspects of rehabilitation policy and provision and monitor how they have discharged their human rights responsibilities towards those most disadvantaged, including people with disability. This article describes the process of developing an expert guided indicator framework to assess governments' efforts and progress in strengthening rehabilitation in line with the Convention on the Rights of Persons with Disabilities.

METHODS

A systems methodology - concept mapping - was used to capture, aggregate and confirm the knowledge of diverse stakeholders on measures thought to be useful for monitoring the implementation of the Convention with respect to health related rehabilitation. Fifty-six individuals generated a list of 107 indicators through online brainstorming which were subsequently sorted by 37 experts from the original panel into non overlapping categories. Forty-one participants rated the indicators for importance and feasibility. Multivariate statistical techniques where used to explore patterns and themes in the data and create the indicators' organizing framework which was verified and interpreted by a select number of participants.

RESULTS

A concept map of 11 clusters of indicators emerged from the analysis grouped into three broader themes: Governance and Leadership (3 clusters); Service Delivery, Financing and Oversight (6 clusters); and Human Resources (2 clusters). The indicator framework was comprehensive and well aligned with the Convention. On average, there was a moderately positive correlation between importance and feasibility of the indicators (r = .58) with experts prioritizing the indicators contained in the clusters of the Governance and Leadership domain. Two of the most important indicators arose from the Service Delivery, Financing and Oversight domain and reflect the need to monitor unmet needs and barriers in access to rehabilitation. In total, 59 indicators achieved above average score for importance and comprised the two-tiered priority set of indicators.

CONCLUSION

Concept mapping was successful in generating a shared model that enables a system's view of the most critical legal, policy and programmatic factors that must be addressed when assessing country efforts to reform, upscale and improve rehabilitation services. The Rehabilitation Systems Diagnosis and Dialogue framework provides a data driven basis for the development of standardized data collection tools to facilitate comparative analysis of rehabilitation systems. Despite agreement on the importance and feasibility of 59 indicators, further research is needed to appraise the applicability and utility of the indicators and secure a realistic assessment of rehabilitation systems.

Behavioural and emotional concerns reported by parents of children attending a neurodevelopmental diagnostic centre

BACKGROUND

Behavioural and emotional problems are a salient concern for parents of children with neurodisability, but little is known about the nature of such concerns in this population, nor about the distribution of concern types across ostensibly different subpopulations.

METHODS

Information about behavioural and emotional concerns was extracted from clinical reports of developmental paediatricians (N = 12) who had assessed children aged 3 to 8 years (N = 129) through three clinics at a major developmental and rehabilitation service centre. All concerns were captured at a granular level. A two-stage, consensus-based interdisciplinary concept-sorting technique was used to identify and group thematically related behavioural and emotional concerns into First Stage Groupings, intended to preserve detail and specificity, and a reduced number of Second Stage Clusters.

RESULTS

A total of 669 discrete concerns were encountered, aggregated to 58 First Stage Concern Groupings and 28 Second Stage Concern Clusters. Findings of the salience of Groupings related to Attention, Concentration and Distractibility, and Anxiousness, Shyness, and Emotional Sensitivity reflect existing literature for children with neurodevelopmental concerns. "Social Isolation/Peer Engagement," "Tantrums/Outbursts/Meltdowns" and "Volatility/Self-regulation Difficulties," and "Sensory Issues" emerged as areas of significant concern and salience as well. Across clinics, three Clusters recurred among the top five observed for each clinic: "Tantrums/Outbursts/Meltdowns," "Inflexibility/Gets Stuck or Fixated," and "Social Behaviours."

CONCLUSIONS

This rich descriptive dataset affords insight into the phenomenology of behaviour and emotional concerns in the daily lives of parents whose children have known or suspected neurodisability. Study findings can inform and sensitize clinicians working with this population. Usefulness is enhanced by inclusion of behavioural material that is subthreshold for a formal psychopathologic diagnosis. Certain concern types are encountered commonly across different clinical subpopulations, supporting a noncategorical view of behavioural and emotional problems as functional attributes that cross neurodisability diagnostic categories.

Quality indicators for responsible use of medicines: a systematic review

OBJECTIVE

All healthcare systems require valid ways to evaluate service delivery. The objective of this study was to identify existing content validated quality indicators (QIs) for responsible use of medicines (RUM) and classify them using multiple frameworks to identify gaps in current quality measurements.

DESIGN

Systematic review without meta-analysis.

SETTING

All care settings.

SEARCH STRATEGY

CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched up to April 2018. An internet search was also conducted. Articles were included if they described medication-related QIs developed using consensus methods. Government agency websites listing QIs for RUM were also included.

ANALYSIS

Several multidimensional frameworks were selected to assess the scope of QI coverage. These included Donabedian's framework (structure, process and outcome), the Anatomical Therapeutic Chemical (ATC) classification system and a validated classification for causes of drug-related problems (c-DRPs; drug selection, drug form, dose selection, treatment duration, drug use process, logistics, monitoring, adverse drug reactions and others).

RESULTS

2431 content validated QIs were identified from 131 articles and 5 websites. Using Donabedian's framework, the majority of QIs were process indicators. Based on the ATC code, the largest number of QIs pertained to medicines for nervous system (ATC code: N), followed by anti-infectives for systemic use (J) and cardiovascular system (C). The most common c-DRPs pertained to 'drug selection', followed by 'monitoring' and 'drug use process'.

CONCLUSIONS

This study was the first systematic review classifying QIs for RUM using multiple frameworks. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. Although many QIs were identified, this approach allowed for the identification of gaps in quality measurement of RUM. In order to more effectively evaluate the extent to which RUM has been achieved, further development of QIs may be required.

The comparative and objective measurement of health promotion capacity-building: from conceptual framework to operationalization

The aim of this study was to analyse and test a theoretical generic health promotion capacity-building framework with empirical data on primary health care. The framework consists of seven dimensions: commitment, management, monitoring and needs assessment, resources, common practices, participation and other core functions. The data were collected in 2014 from all the health centres in Finland, of which 156 (99%) submitted their data. The data were scored by the quality of activities on a scale from 0 to 100, where 100 stands for desirable quality. Individual indicators were nested into subdimensions, which in turn were nested into the dimensions of the theoretical framework. Variables were clustered using the dimensions and subdimensions as initial partitions. The internal consistency of dimensions and subdimensions was tested with standardized Cronbach's alpha both before and after the clustering analysis. The results showed that although the internal consistency of the dimensions was high in the initial classification, it is possible to get even more consistent dimensions. The internal consistency of the initial classification varied from 0.62 in participation to 0.93 in common practices. In the clustering analysis, 45 out of 203 indicators were assigned to a dimension different from the initial partition. The biggest gain in internal consistency was achieved in the subdimension of systematic mass communications by relocating two indicators. This study suggests that it is possible to assess the health promotion capacity-building of organizations in a coherent way with comparable and objective indicators. These analyses also show that the number of indicators can be reduced. It would be interesting to see how the framework works in other governmental structures or political contexts.

[Assessing the balance of quality indicator sets of external quality assurance according to SGB V section 136]

INTRODUCTION

The value and usefulness of the results of indicator-based performance measurement in healthcare for different purposes do not only depend on the methodological quality of the individual indicators but also on the composition of the indicator sets. So far, the balance of the currently used indicator sets of the German mandatory national performance measurement system for hospitals has not been systematically analyzed.

MATERIAL AND METHODS

Due to the lack of a methodological gold standard for the assessment of balance and orientation of indicator sets we adapted the OECD concept of quality dimensions and defined four categories: 1) "Achieving primary goals of treatment", 2) "Avoiding adverse events", 3) "Indication" and 4) "Patient-centeredness". We defined rules for the assignment to the categories and analyzed the distribution of the 239 indicators from 29 medical areas in relation to these categories.

RESULTS

63 indicators (26.4 %) were assigned to the category "Achieving primary goals of treatment", 153 (64.0 %) to the category "Avoiding adverse events", 18 (7.5 %) to the category "Indication", one indicator (0.4 %) to the category "Patient-centeredness". Four indicators (1.7 %) addressed documentation quality. 12 of the 29 indicator sets only covered one OECD quality dimension by at least one indicator.

CONCLUSIONS

The current indicator sets seem to be unbalanced with a strong focus on the category "Avoiding adverse events". As regards the goal of monitoring compliance with minimal safety standards and performing improvement interventions, the direction of the indicator sets seems to be appropriate. With respect to other goals, such as for example the identification of "excellence", further development efforts are required. One relevant reason for the dominant focus on the category "Avoiding adverse events" seems to be that data sources for a follow-up and for the inclusion of the patient perspective have not been available until recently. There is a strong demand for the consequent use of these data sources to optimize the interpretability and value of the current performance measurement. The methodological approach presented may offer useful information to assess the value of indicator sets for different purposes although further development and research is necessary.

Performance measures for a dialysis setting

OBJECTIVES

This study from Japan extracted performance measures for dialysis unit management and investigated their characteristics from professional views.

METHODS

Two surveys were conducted using self-administered questionnaires, in which dialysis managers/staff were asked to rate the usefulness of 44 performance indicators. A total of 255 managers and 2,097 staff responded.

RESULTS

Eight performance measures were elicited from dialysis manager and staff responses: these were safety, operational efficiency, quality of working life, financial effectiveness, employee development, mortality, patient/employee satisfaction and patient-centred health care. These performance measures were almost compatible with those extracted in overall healthcare settings in a previous study. Internal reliability, content and construct validity of the performance measures for the dialysis setting were ensured to some extent. As a general trend, both dialysis managers and staff perceived performance measures as highly useful, especially for safety, mortality, operational efficiency and patient/employee satisfaction, but showed relatively low concerns for patient-centred health care and employee development. However, dialysis managers' usefulness perceptions were significantly higher than staff.

CONCLUSIONS

Important guidelines for designing a holistic hospital/clinic management system were yielded. Performance measures must be balanced for outcomes and performance shaping factors (PSF); a common set of performance measures could be applied to all the healthcare settings, although performance indicators of each measure should be composed based on the application field and setting; in addition, sound causal relationships between PSF and outcome measures/indicators should be explored for further improvement.

Medical directors’ perspectives on strengthening hospital quality and safety

Purpose

The purpose of this paper is to understand the concerns and factors that impact on hospital quality and safety, particularly related to use of performance data, within a setting of devolved governance.

Design/methodology/approach

This qualitative study used thematic analysis of interviews with public hospital medical directors. For additional context, findings were framed by themes from a review of hospital safety and quality in the same jurisdiction.

Findings

Varying approaches and levels of complexity were described about what and how performance data are reviewed, prioritised, and quality improvements implemented. Although no consistent narrative emerged, facilitators of improvement were suggested relating to organisational culture, governance, resources, education, and technologies. These hospital-level perspectives articulate with and expand on the system-level themes in a state-wide review of hospital safety and quality.

Research limitations/implications

The findings are not generalisable, but point to an underlying absence of system-wide agreement on how to perceive, retrieve, analyse, prioritise and action hospital performance data.

Practical implications

Lack of electronic medical records and an inefficient incident reporting system limits the extent to which performance and incident data can be analysed, linked and shared, thus limiting hospital performance improvement, oversight and learning.

Social implications

Variable approaches to quality and safety, standards of care, and hospital record keeping and reporting, mean that healthcare consumers might expect inconsistency across Victorian hospitals.

Originality/value

The views of medical directors have been little researched. This work uses their voice to better understand contextual factors that situate and impact on hospital quality and safety towards understanding the mixed effectiveness of hospital quality improvement strategies.

Quality of Midwife-provided Intrapartum Care in Amhara Regional State, Ethiopia

BACKGROUND

Despite much progress recently, Ethiopia remains one of the largest contributors to the global burden of maternal and newborn deaths and stillbirths. Ethiopia's plan to meet the sustainable development goals for maternal and child health includes unprecedented emphasis on improving quality of care. The purpose of this study was to assess the quality of midwifery care during labor, delivery and immediate postpartum period.

METHODS

A cross-sectional study using multiple data collection methods and a 2-stage cluster sampling technique was conducted from January 25 to February 14, 2015 in government health facilities of the Amhara National Regional State of Ethiopia. Direct observation of performance was used to determine competence of midwives in providing care during labor, delivery, and the first 6 h after childbirth. Inventory of drugs, medical equipment, supplies, and infrastructure was conducted to identify availability of resources in health facilities. Structured interview was done to assess availability of resources and performance improvement opportunities. Data analysis involved calculating percentages, means and chi-square tests.

RESULTS

A total of 150 midwives and 56 health facilities were included in the study. The performance assessment showed 16.5% of midwives were incompetent, 72.4% were competent, and 11.1% were outstanding in providing routine intrapartum care. Forty five midwives were observed while managing 54 obstetric and newborn complications and 41 (91%) of them were rated competent. Inventory of resources found that the proportion of facilities with more than 75% of the items in each category was 32.6% for drugs, 73.1% for equipment, 65.4% for supplies, 47.9% for infection prevention materials, and 43.6% for records and forms. Opportunities for performance improvement were inadequate, with 31.3% reporting emergency obstetric and newborn care training, and 44.7% quarterly or more frequent supportive supervision. Health centers fared worse in provider competence, physical resources, and quality improvement practices except for supportive supervision visits and in-service training.

CONCLUSIONS

Although our findings indicate most midwives are competent in giving routine and emergency intrapartum care, the major gaps in the enabling environment and the significant proportion of midwives with unsatisfactory performance suggest that the conditions for providing quality intrapartum care are not optimal.

Stakeholder driven indicators for eHealth performance management

The goal of the present article is to compile a corpus of indicators of eHealth development evaluation that would essentially reflect stakeholder approaches and complement technical indicators of assessment of an eHealth system. Consequently, the assessment of the development of an eHealth system would reflect stakeholder approaches and become an innovative solution in attempting to improve productivity of IT projects in the field of health care. The compiled minimum set of indicators will be designed to monitor implementation of the national eHealth information system. To ensure reliability of the quality research, the respondents were grouped in accordance to the geographical distribution and diversity of the levels and types of the represented jobs and institutions. The applied analysis implies several managerial insights on the hierarchy of eHealth indicators. These insights may be helpful in recommending priority activities in implementation of an eHealth data system on the national or international level. The research is practically useful as it is the first to deal with the topic in Lithuania and its theoretical and practical aspect are particularly relevant in implementation of an eHealth data system in Lithuania. The eHealth assessment indicators presented in the article may be practically useful in two aspects: (1) as key implementation guidelines facilitating the general course of eHealth system development and (2) as a means to evaluate eHealth outcomes.

Measuring healthcare productivity - from unit to system level

Purpose - Healthcare productivity is a growing issue in most Western countries where healthcare expenditure is rapidly increasing. Therefore, accurate productivity metrics are essential to avoid sub-optimization within a healthcare system. The purpose of this paper is to focus on healthcare production system productivity measurement. Design/methodology/approach - Traditionally, healthcare productivity has been studied and measured independently at the unit, organization and system level. Suggesting that productivity measurement should be done in different levels, while simultaneously linking productivity measurement to incentives, this study presents the challenges of productivity measurement at the different levels. The study introduces different methods to measure productivity in healthcare. In addition, it provides background information on the methods used to measure productivity and the parameters used in these methods. A pilot investigation of productivity measurement is used to illustrate the challenges of measurement, to test the developed measures and to prove the practical information for managers. Findings - The study introduces different approaches and methods to measure productivity in healthcare. Practical implications - A pilot investigation of productivity measurement is used to illustrate the challenges of measurement, to test the developed measures and to prove the practical benefits for managers. Originality/value - The authors focus on the measurement of the whole healthcare production system and try to avoid sub-optimization. Additionally considering an individual patient approach, productivity measurement is examined at the unit level, the organizational level and the system level.

Achieving value for money in health: a comparative analysis of OECD countries and regional countries

OBJECTIVE

To measure efficiency gains in health sector over the years 1995 to 2013 in OECD, EU, non-member European countries.

METHODS

An output-oriented DEA model with variable return to scale, and residuals estimated by regression equations were used to estimate efficiencies of health systems. Slacks for health care outputs and inputs were calculated by using DEA multistage method of estimating country efficiency scores.

RESULTS

Better health outcomes of countries were related with higher efficiency. Japan, France, or Sweden were found to be peer-efficient countries when compared to other developed countries like Germany and United States. Increasing life expectancy beyond a certain high level becomes very difficult to achieve. Despite declining marginal productivity of inputs on health outcomes, some developed countries and developing countries were found to have lowered their inefficiencies in the use of health inputs. Although there was no systematic relationship between political system of countries and health system efficiency, the objectives of countries on social and health policy and the way of achieving these objectives might be a factor increasing the efficiency of health systems.

CONCLUSIONS

Economic and political stability might be as important as health expenditure in improving health system goals. A better understanding of the value created by health expenditures, especially in developed countries, will require analysis of specific health interventions that can increase value for money in health. Copyright © 2016 John Wiley & Sons, Ltd.

Performance indicators: healthcare professionals’ views

Purpose

– The purpose of this paper is to capture factors behind professional views of indicator usefulness as a common structure for assessing healthcare performance and their important characteristics to design limited key performance indicators (PIs) for holistic hospital management.

Design/methodology/approach

– Two surveys were conducted using self-administered questionnaires, in which hospital manager/staff respondents were asked to rate the 52 PIs’ usefulness. In total, 228 manager and 894 staff responses were collected.

Findings

– Eight factors were elicited from manager and staff responses as performance measures with 72 percent cumulative variance accounted for. Hospital managers and staff showed similar performance measure perceptions: high-utility acknowledgment on safety, operational efficiency and patient/employee satisfaction but relatively low-employee development concerns. Manager indicator usefulness perceptions were rather homogeneous and significantly higher than staff for almost all performance measures.

Practical implications

– Homogeneous manager views mean that a single key PI set for hospital management may be established regardless of hospital attributes. The following aspects may be measures that should be managed in a healthcare organization based on their key PIs: patient/employee safety, operational efficiency, financial effectiveness and patient/employee satisfaction.

Originality/value

– This is a pilot study on hospital management PIs in Japan. The eight-dimensional factor structure and findings about healthcare provider perceptions may be useful for healthcare management.

Measuring and improving quality in university hospitals in Canada: The Collaborative for Excellence in Healthcare Quality

Measuring and monitoring overall health system performance is complex and challenging but is crucial to improving quality of care. Today's health care organizations are increasingly being held accountable to develop and implement actions aimed at improving the quality of care, reducing costs, and achieving better patient-centered care. This paper describes the development of the Collaborative for Excellence in Healthcare Quality (CEHQ), a 5-year initiative to achieve higher quality of patient care in university hospitals across Canada. This bottom-up initiative took place between 2010 and 2015, and was successful in engaging health care leaders in the development of a common framework and set of performance measures for reporting and benchmarking, as well as working on initiatives to improve performance. Despite its successes, future efforts are needed to provide clear national leadership on standards for measuring performance.

Use of Balanced Scorecard Methodology for Performance Measurement of the Health Extension Program in Ethiopia

In 2004, Ethiopia introduced a community-based Health Extension Program to deliver basic and essential health services. We developed a comprehensive performance scoring methodology to assess the performance of the program. A balanced scorecard with six domains and 32 indicators was developed. Data collected from 1,014 service providers, 433 health facilities, and 10,068 community members sampled from 298 villages were used to generate weighted national, regional, and agroecological zone scores for each indicator. The national median indicator scores ranged from 37% to 98% with poor performance in commodity availability, workforce motivation, referral linkage, infection prevention, and quality of care. Indicator scores showed significant difference by region (P < 0.001). Regional performance varied across indicators suggesting that each region had specific areas of strength and deficiency, with Tigray and the Southern Nations, Nationalities and Peoples Region being the best performers while the mainly pastoral regions of Gambela, Afar, and Benishangul-Gumuz were the worst. The findings of this study suggest the need for strategies aimed at improving specific elements of the program and its performance in specific regions to achieve quality and equitable health services.

A Scoping Review of Clinical Practice Improvement Methodology Use in Rehabilitation

Context

The Clinical Practice Improvement (CPI) approach is a methodological and quality improvement approach that has emerged and is gaining in popularity. However, there is no systematic description of its use or the determinants of its practice in rehabilitation settings.

Method

We performed a scoping review of the use of CPI methodology in rehabilitation settings.

Results

A total of 103 articles were reviewed. We found evidence of 13 initiatives involving CPI with six different populations. A total of 335 citations of determinants were found, with 68.7% related to CPI itself. Little information was found about what type of external and internal environment, individual characteristics and implementation process might facilitate or hinder the use of CPI.

Conclusion

Given the growing popularity of this methodological approach, CPI initiatives would gain from increasing knowledge of the determinants of its success and incorporating them in future implementation.

Assessment of the pregnancy education programme with 'EDUMA2' questionnaire in Madrid (Spain)

RATIONALE, AIMS AND OBJECTIVES

The prenatal education promotes the empowerment of parents during pregnancy and postnatal period. This study aimed to assess the quality of educational sessions held in the third trimester of pregnancy as part of the parenting education programme for Spanish National Health System in Madrid.

METHODS

The design is a cross-sectional study in 41 primary care centres in the autonomous community of Madrid, which is one of the 17 autonomous communities that constitute the Spanish State, each wick medical responsibilities. The participants are a representative probability sample of 928 attendees to the programme. The assessment instrument is 'EDUMA2' questionnaire (Cronbach's alpha = 0.829) of 56 variables. Descriptive statistical analysis was performed using SPSS. The project was approved by the Research and Ethics Committees of the University Hospital of La Paz.

RESULTS

The uptake efficiency immigration risk is 14.7%, and lack of social support is 8.7%. The functionality in organization, teaching and methodology is high in 90.5%. The learning effectiveness of health habits, care and techniques is significant and greater than 60% in the 14 parameters studied. Satisfaction is very high at 67.5%. The immediate impact in terms of control or safety increase is significant and greater than 71% and significantly greater than 40% and for increasing the bonding with the baby.

CONCLUSIONS

No jobs found with which to compare. The assessment of the programme with adequate psychometric characteristics questionnaire allows designing strategies and research to improve the quality of prenatal education.

Teaching hospital performance: towards a community of shared values?

This paper explores the performance dimensions of Italian teaching hospitals (THs) by considering the multiple constituent model approach, using measures that are subjective and based on individual ideals and preferences. Our research replicates a study of a French TH and deepens it by adjusting it to the context of an Italian TH. The purposes of this research were as follows: to identify emerging views on the performance of teaching hospitals and to analyze how these views vary among hospital stakeholders. We conducted an in-depth case study of a TH using a quantitative survey method. The survey uses a questionnaire based on Parsons' social system action theory, which embraces the major models of organizational performance and covers three groups of internal stakeholders: physicians, caregivers and administrative staff. The questionnaires were distributed between April and September 2011. The results confirm that hospital performance is multifaceted and includes the dimensions of efficiency, effectiveness and quality of care, as well as organizational and human features. There is a high degree of consensus among all observed stakeholder groups about these values, and a shared view of performance is emerging. Our research provides useful information for defining management priorities to improve the performance of THs.

Measuring quality of care: considering measurement frameworks and needs assessment to guide quality indicator development

OBJECTIVE

In this article, we describe one approach for evaluating the value of developing quality indicators (QIs).

STUDY DESIGN AND SETTING

We focus on describing how to develop a conceptual measurement framework and how to evaluate the need to develop QIs. A recent process to develop QIs for injury care is used for illustration.

RESULTS

Key steps to perform before developing QIs include creating a conceptual measurement framework, determining stakeholder perspectives, and performing a QI needs assessment. QI development is likely to be most beneficial for medical problems for which quality measures have not been previously developed or are inadequate and that have a large burden of illness to justify quality measurement and improvement efforts, are characterized by variable or substandard care such that opportunities for improvement exist, and have evidence that improving quality of care will improve patient health.

CONCLUSION

By developing a conceptual measurement framework and performing a QI needs assessment, developers and users of QIs can target their efforts.

Avaliação da qualidade do cuidado hospitalar no Brasil: uma revisão sistemática

A avaliação da qualidade de serviços hospitalares vem ganhando importância no mundo, sendo impulsionada pela demanda de financiadores, prestadores, profissionais e pacientes. O objetivo deste estudo é revisar a literatura sobre estudos de avaliação da qualidade hospitalar no Brasil e analisar as principais abordagens, metodologias e indicadores utilizados. Foi aplicada revisão sistemática de artigos científicos, dissertações e teses com análises empíricas sobre o tema, publicados entre 1990 e 2011. Foram identificados 2.169 documentos e incluídos 62 documentos na revisão, que representam 48 estudos distintos. Predominou o uso de fontes de dados secundárias com análise das dimensões efetividade, adequação, segurança e eficiência, destacando-se a aplicação da taxa de mortalidade, taxa de adequação, taxa de eventos adversos e tempo de permanência. Métodos que controlam diferenças de risco dos pacientes foram majoritariamente aplicados. Busca-se com esta revisão contribuir apontando elementos centrais para o desenvolvimento do tema no país e para a qualificação do cuidado prestado.

Measuring the quality of a childhood cancer care delivery system: quality indicator development

OBJECTIVES

A set of indicators to assess the quality of a childhood cancer system has not been identified in any jurisdiction internationally, despite the movement toward increased accountability and provision of high-quality care with limited health care resources. This study was conducted to develop a set of quality indicators (QIs) of a childhood cancer control and health care delivery system in Ontario, Canada.

METHODS

A systematic review and targeted gray literature search were conducted to identify potential childhood cancer QIs. A series of investigator focus group sessions followed to review all QIs identified in the literature, and to generate a provisional QI set for a childhood cancer system. QIs were evaluated by three content experts in a sequential selection process on the basis of a series of criteria to select a subset for presentation to stakeholders. Following an appraisal of the relevance of quality assessment frameworks, remaining QIs were mapped onto the Cancer System Quality Index framework.

RESULTS

The systematic review yielded few relevant childhood cancer system QIs. Overall, 120 provisional QIs were developed by the investigator group. Based on median QI rating scores, representation across the childhood cancer continuum, and feasibility of data collection, a subset of 33 QIs was selected for stakeholder consideration.

CONCLUSIONS

The subset of 33 QIs developed on the basis of a systematic literature review and consensus provides the basis for the selection of a set of QIs for ongoing, standardized monitoring of various dimensions of quality in a childhood cancer system.

[Evaluation of performance of health systems: a model for analysis]

This paper presents a review of the Dimension Matrix for Evaluation of the Brazilian Health System that was initially developed in 2003, as well as a conceptual update of some of the sub-dimensions for the evaluation of health service performance, namely effectiveness, access, efficiency and appropriateness of health care. It also describes the indicator selection process as well as the results obtained in each performance dimension. The behavior of the indicators used to assess the performance of health services in Brazil, with respect to each sub-dimension, was not uniform. Areas of marked improvement were found in indicators that are influenced by activities in the field of primary care. The most significant improvements were seen in the sub-dimensions of Effectiveness and Access. With respect to the Efficiency of health services, situations of high efficiency coexist with others with substandard performance. The performance of health services in the sub-dimension of Appropriateness of Health Care was the lowest of all indicators.

Cluster-randomized trial to evaluate the effects of a quality improvement program on management of non-ST-elevation acute coronary syndromes: The European Quality Improvement Programme for Acute Coronary Syndromes (EQUIP-ACS)

BACKGROUND

Registries have shown that quality of care for acute coronary syndromes (ACS) often falls below the standards recommended in professional guidelines. Quality improvement (QI) is a strategy to improve standards of clinical care for patients, but the efficacy of QI for ACS has not been tested in randomized trials.

METHODS

We undertook a prospective, cluster-randomized, multicenter, multinational study to evaluate the efficacy of a QI program for ACS. Participating centers collected data on consecutive admissions for non-ST-elevation ACS for 4 months before the QI intervention and 3 months after. Thirty-eight hospitals in France, Italy, Poland, Spain, and the United Kingdom were randomized to receive the QI program or not, 19 in each group. We measured 8 in-hospital quality indicators (risk stratification, coronary angiography, anticoagulation, β-blockers, statins, angiotensin-converting enzyme inhibitors, and clopidogrel loading and maintenance) before and after the intervention and compared composite changes between the QI and non-QI groups.

RESULTS

A total of 2604 patients were enrolled. The absolute overall change in use of quality indicators in the QI group was 8.5% compared with 0.8% in the non-QI group (odds ratio for achieving a quality indicator in QI versus non-QI 1.66, 95% CI 1.43-1.94; P < .001). The main changes were observed in the use of risk stratification and clopidogrel loading dose.

CONCLUSIONS

The QI strategy resulted in a significant improvement in the quality indicators measured. This type of QI intervention can lead to useful changes in health care practice for ACS in a wide range of settings.

Scorecards for health system performance assessment: the New Zealand example

OBJECTIVE

To develop a national scorecard for assessing health system performance derived from routine data.

METHODS

We drew upon national and international data to develop benchmarks for health system performance, then applied basic ratio scores to compare New Zealand performances to the benchmark. 64 indicators were included in four assessment categories: healthy lives, quality, access, and efficiency. In a fifth category, 27 of these indicators were used to score health system equity. Indicator scores in each category were then averaged to give a health system score out of 100.

RESULTS

New Zealand's health system achieved an overall score of 71 out of 100. The system scored relatively well on quality and efficiency, but poorly on equity despite considerable government investment in reducing inequalities.

CONCLUSIONS

The scorecard offers a useful method for combining a range of data to give an overall picture of health system performance, highlighting strengths, weaknesses and areas for improvement. This initial study provides a baseline for assessing New Zealand's performance over time and, where data permit, a template for other countries to follow.

Configuring balanced scorecards for measuring health system performance: evidence from 5 years' evaluation in Afghanistan

BACKGROUND

In 2004, Afghanistan pioneered a balanced scorecard (BSC) performance system to manage the delivery of primary health care services. This study examines the trends of 29 key performance indicators over a 5-year period between 2004 and 2008.

METHODS AND FINDINGS

Independent evaluations of performance in six domains were conducted annually through 5,500 patient observations and exit interviews and 1,500 provider interviews in >600 facilities selected by stratified random sampling in each province. Generalized estimating equation (GEE) models were used to assess trends in BSC parameters. There was a progressive improvement in the national median scores scaled from 0-100 between 2004 and 2008 in all six domains: patient and community satisfaction of services (65.3-84.5, p<0.0001); provider satisfaction (65.4-79.2, p<0.01); capacity for service provision (47.4-76.4, p<0.0001); quality of services (40.5-67.4, p<0.0001); and overall vision for pro-poor and pro-female health services (52.0-52.6). The financial domain also showed improvement until 2007 (84.4-95.7, p<0.01), after which user fees were eliminated. By 2008, all provinces achieved the upper benchmark of national median set in 2004.

CONCLUSIONS

The BSC has been successfully employed to assess and improve health service capacity and service delivery using performance benchmarking during the 5-year period. However, scorecard reconfigurations are needed to integrate effectiveness and efficiency measures and accommodate changes in health systems policy and strategy architecture to ensure its continued relevance and effectiveness as a comprehensive health system performance measure. The process of BSC design and implementation can serve as a valuable prototype for health policy planners managing performance in similar health care contexts. Please see later in the article for the Editors' Summary.

Do inter-hospital comparisons of in-hospital, acute myocardial infarction case-fatality rates serve the purpose of fostering quality improvement? An evaluative study

BACKGROUND

In-hospital case-fatality rates in patients, admitted for acute myocardial infarction (AMI-CFRs), are internationally used as a quality indicator. Attempting to encourage the hospitals to assume responsibility, the Belgian Ministry of Health decided to stimulate initiatives of quality improvement by means of a limited set of indicators, among which AMI-CFR, to be routinely analyzed. In this study we aimed, by determining the existence of inter-hospital differences in AMI-CFR, (1) to evaluate to which extent Belgian discharge records allow the assessment of quality of care in the field of AMI, and (2) to identify starting points for quality improvement.

METHODS

Hospital discharge records from all the Belgian short-term general hospitals in the period 2002-2005. The study population (N = 46,287) included patients aged 18 years and older, hospitalized for AMI. No unique patient identifier being present, we tried to track transferred patients. We assessed data quality through a comparison of MCD with data from two registers for acute coronary events and through transfer and sensitivity analyses. We compared AMI-CFRs across hospitals, using multivariable logistic regression models. In the main model hospitals, Charlson's co-morbidity index, age, gender and shock constituted the covariates. We carried out two types of analyses: a first one wherein transferred-out cases were excluded, to avoid double counting of patients when computing rates, and a second one with exclusion of all transferred cases, to allow the study of patients admitted into, treated in and discharged from the same hospital.

RESULTS

We identified problems regarding both the CFR's numerator and denominator.Sensitivity analyses revealed differential coding and/or case management practices. In the model with exclusion of transfer-out cases, the main determinants of AMI-CFR were cardiogenic shock (OR(adj) 23.0; 95% CI [20.9;25.2]), and five-year age groups OR(adj) 1.23; 95% CI [1.11;1.36]). Sizable inter-hospital and inter-type of hospital differences {(OR(comunity vs tertiary hospitals)1.36; 95% CI [1.34;1.39]) and (OR(intermediary vs tertiary hospitals)1.36; 95% CI [1.34;1.39])}, and nonconformities to guidelines for treatment were observed.

CONCLUSIONS

Despite established data quality shortcomings, the magnitude of the observed differences and the nonconformities constitute leads to quality improvement. However, to measure progress, ways to improve and routinely monitor data quality should be developed.