Health system frameworks and performance indicators in eight countries: A comparative international analysis

Health system description and assessment: a scoping review of templates for systematic analyses

BACKGROUND

Understanding and comparing health systems is key for cross-country learning and health system strengthening. Templates help to develop standardised and coherent descriptions and assessments of health systems, which then allow meaningful analyses and comparisons. Our scoping review aims to provide an overview of existing templates, their content and the way data is presented.

MAIN BODY

Based on the WHO building blocks framework, we defined templates as having (1) an overall framework, (2) a list of indicators or topics, and (3) instructions for authors, while covering (4) the design of the health system, (5) an assessment of health system performance, and (6) should cover the entire health system. We conducted a scoping review of grey literature published between 2000 and 2023 to identify templates. The content of the identified templates was screened, analyzed and compared. We found 12 documents that met our inclusion criteria. The building block `health financing´ is covered in all 12 templates; and many templates cover ´service delivery´ and ´health workforce'. Health system performance is frequently assessed with regard to 'access and coverage', 'quality and safety', and 'financial protection'. Most templates do not cover 'responsiveness' and 'efficiency'. Seven templates combine quantitative and qualitative data, three are mostly quantitative, and two are primarily qualitative. Templates cover data and information that is mostly relevant for specific groups of countries, e.g. a particular geographical region, or for high or for low and middle-income countries (LMICs). Templates for LMICs rely more on survey-based indicators than administrative data.

CONCLUSIONS

This is the first scoping review of templates for standardized descriptions of health systems and assessments of their performance. The implications are that (1) templates can help analyze health systems across countries while accounting for context; (2) template-guided analyses of health systems could underpin national health policies, strategies, and plans; (3) organizations developing templates could learn from approaches of other templates; and (4) more research is needed on how to improve templates to better achieve their goals. Our findings provide an overview and help identify the most important aspects and topics to look at when comparing and analyzing health systems, and how data are commonly presented. The templates were created by organizations with different agendas and target audiences, and with different end products in mind. Comprehensive health systems analyses and comparisons require production of quantitative indicators and complementing them with qualitative information to build a holistic picture.

CLINICAL TRIAL REGISTRATION

  Not applicable.

A multivariable analysis to predict variations in hospital mortality using systems-based factors of healthcare delivery to inform improvements to healthcare design within the English NHS

Over the last decade, the strain on the English National Health Service (NHS) has increased. This has been especially felt by acute hospital trusts where the volume of admissions has steadily increased. Patient outcomes, including inpatient mortality, vary between trusts. The extent to which these differences are explained by systems-based factors, and whether they are avoidable, is unclear. Few studies have investigated these relationships. A systems-based methodology recognises the complexity of influences on healthcare outcomes. Rather than clinical interventions alone, the resources supporting a patient's treatment journey have near-equal importance. This paper first identifies suitable metrics of resource and demand within healthcare delivery from routinely collected, publicly available, hospital-level data. Then it proceeds to use univariate and multivariable linear regression to associate such systems-based factors with standardised mortality. Three sequential cross-sectional analyses were performed, spanning the last decade. The results of the univariate regression analyses show clear relationships between five out of the six selected predictor variables and standardised mortality. When these five predicators are included within a multivariable regression analysis, they reliably explain approximately 36% of the variation in standardised mortality between hospital trusts. Three factors are consistently statistically significant: the number of doctors per hospital bed, bed occupancy, and the percentage of patients who are placed in a bed within four hours after a decision to admit them. Of these, the number of doctors per bed had the strongest effect. Linear regression assumption testing and a robustness analysis indicate the observations have internal validity. However, our empirical strategy cannot determine causality and our findings should not be interpreted as established causal relationships. This study provides hypothesis-generating evidence of significant relationships between systems-based factors of healthcare delivery and standardised mortality. These have relevance to clinicians and policymakers alike. While identifying causal relationships between the predictors is left to the future, it establishes an important paradigm for further research.

How can strategies based on performance measurement and feedback support changes in nursing practice? A theoretical reflection drawing on Habermas' social perspective

Strategies based on performance measurement and feedback are commonly used to support quality improvement among nurses. These strategies require practice change, which, for nurses, rely to a large extent on their capacity to coordinate with each other effectively. However, the levers for coordinated action are difficult to mobilize. This discussion paper offers a theoretical reflection on the challenges related to coordinating nurses' actions in the context of practice changes initiated by performance measurement and feedback strategies. We explore how Jürgen Habermas' theory of Communicative Action may shed light on the issues underlying nurses' collective actions and self-determination in practice change and the implications for the design of strategies based on performance measurement and feedback. Based on this theory, we propose differences between communicative and functional coordination according to the nature of the actions and the purposes involved. The domains of action underlying these coordination processes, which Habermas referred to as the lifeworld and the system, are then used to draw a parallel with aspects of nursing practice. Further exploration of these concepts allows us to consider the tensions between the demands of the system and the self-determination of nurses within their practice.

Development of Data Quality Indicators for Improving Hospital International Classification of Diseases-coded Health Data Quality Globally

BACKGROUND

Hospital inpatient data, coded using the International Classification of Diseases (ICD), is widely used to monitor diseases, allocate resources and funding, and evaluate patient outcomes. As such, hospital data quality should be measured before use; however, currently, there is no standard and international approach to assess ICD-coded data quality.

OBJECTIVE

To develop a standardized method for assessing hospital ICD-coded data quality that could be applied across countries: Data quality indicators (DQIs).

RESEARCH DESIGN

To identify a set of candidate DQIs, we performed an environmental scan, reviewing gray and academic literature on data quality frameworks and existing methods to assess data quality. Indicators from the literature were then appraised and selected through a 3-round Delphi process. The first round involved face-to-face group and individual meetings for idea generation, while the second and third rounds were conducted remotely to collect online ratings. Final DQIs were selected based on the panelists' quantitative and qualitative feedback.

SUBJECTS

Participants included international experts with expertise in administrative health data, data quality, and ICD coding.

RESULTS

The resulting 24 DQIs encompass 5 dimensions of data quality: relevance, accuracy and reliability; comparability and coherence; timeliness; and Accessibility and clarity. These will help stakeholders (eg, World Health Organization) to assess hospital data quality using the same standard across countries and highlight areas in need of improvement.

CONCLUSIONS

This novel area of research will facilitate international comparisons of ICD-coded data quality and be valuable to future studies and initiatives aimed at improving hospital administrative data quality.

Data for health system comparison and assessment in the African Region: A review of 63 indicators available in international databases

Background

Achieving universal health coverage in the African region requires health systems strengthening. Assessing and comparing health systems contributes to this process, but requires internationally comparable data. The European Observatory on Health Systems and Policies has produced Health Systems in Transition (HiT) reviews in Europe, Asia, North America and the Caribbean with a standardised template. This study explores data availability in international databases for the quantitative health and health system indicators in the HiT template for the WHO African region.

Methods

We identified ten databases which contained data for 40 of the 80 original HiT indicators and an additional 23 proxy indicators to fill some gaps. We then assessed data availability for the resulting 63 indicators by country and time, i.e. first/last year of data, years of data available overall and since 2000, and we explored for each indicator (1) against the country with the greatest availability overall and (2) against annual availability for all years since 2000.

Results

Overall data availability was greatest in South Africa (93.0% of possible total points) and least in South Sudan (59.5%). Since 2000, Uganda (60.4%) has had the highest data availability and South Sudan (37.2%) the lowest. By topic, data availability was the highest for health financing (91.4%; median start/end date 2000/2019) and background characteristics (88.5%; 1990/2020) and was considerably lower for health system performance (54.5%; 2000/2018) and physical and human resources (44.8%; 2004/2013). Data are available for different years in different countries, and at irregular intervals, complicating time series analysis. No data are available for service provision indicators.

Conclusions

Gaps in data in international databases across time, countries, and topics undermine systematic health systems comparisons and assessments, regional health systems strengthening, and efforts to achieve universal health coverage. More efforts are needed to strengthen national data collection and management and integrate national data into international databases to support cross-country assessments, peer learning, and planning. In tandem, more research is needed to understand the specific historical, cultural, administrative, and technological determinants influencing country data availability, as well as the facilitators and barriers of data sharing between countries and international databases, and the potential of new technologies to increase timeliness of data.

Economic Indicators, Quantity and Quality of Health Care Resources Affecting Post-surgical Mortality

OBJECTIVE

to identify correlations between quality and quantity of health care resources, national economic indicators, and postoperative in-hospital mortality as reported in the EUSOS study.

METHODS

Different variables were identified from a series of publicly available database. Postoperative in-hospital mortality was identified as reported by EUSOS study. Spearman non-parametric and Coefficients of non-linear regression were calculated.

RESULTS

Quality of health care resources was strongly and negatively correlated to postoperative in-hospital mortality. Quantity of health care resources were negatively and moderately correlated to postoperative in-hospital mortality. National economic indicators were moderately and negatively correlated to postoperative in-hospital mortality. General mortality, as reported by WHO, was positively but very moderately correlated with postoperative in-hospital mortality.

CONCLUSIONS

Postoperative in-hospital mortality is strongly determined by quality of health care instead of quantity of health resources and health expenditures. We suggest that improving the quality of health care system might reduce postoperative in-hospital mortality.

[Development of an indicator set for the evaluation of the quality of routine ambulatory health care for common disorders in children and adolescents]

BACKGROUND

In Germany, no consented quality indicator set (QI set) exists to date that can be used to assess the quality of pediatric care. Therefore, the aim of the project "Assessment of the quality of routine ambulatory health care for common disorders in children and adolescents" (QualiPäd) funded by the Innovation Committee of the Federal Joint Committee (grant no.: 01VSF19035) was to develop a QI set for the diseases asthma, atopic eczema, otitis media, tonsillitis, attention-deficit hyperactivity disorder (ADHD), depression and conduct disorder.

METHODS

For the observation period 2018/2019, quality indicators (QIs) were searched in indicator databases, guidelines and literature databases and complemented in part by newly formulated QIs (e.g., derived from guideline recommendations). The QIs were then assigned to content categories and dimensions according to Donabedian and OECD and reduced by removing duplicates. Finally, a panel of experts consulted the QIs using the modified RAND-UCLA Appropriateness Method (RAM).

RESULTS

The search resulted in a preliminary QI set of 2324 QIs. After the reduction steps and the evaluation of the experts, 282 QIs were included in the QI set (asthma: 72 QIs, atopic eczema: 25 QIs, otitis media: 31 QIs, tonsillitis: 12 QIs, ADHD: 53 QIs, depression: 43 QIs, conduct disorder: 46 QIs). The QIs are distributed among the following different categories: Therapy (138 QIs), Diagnostics (95 QIs), Patient-reported outcome measures/Patient-reported experience measures (PROM/PREM) (45 QIs), Practice management (31 QIs), and Health reporting (4 QIs). In the Donabedian model, 89% of the QIs capture process quality, 9% outcome quality, and 2% structural quality; according to the OECD classification, 61% measure effectiveness, 23% patient-centeredness, and 16% safety of care.

CONCLUSION

The consented QI set is currently being tested and can subsequently be used (possibly modified) to measure the quality of routine outpatient care for children and adolescents in Germany, in order to indicate the status quo and potential areas for improvement in outpatient care.

Identifying potential predictable indicators for the management of tertiary hospitals

BACKGROUND

The European University Hospitals Alliance (EUHA) recognises the need to move from the classical approach of measuring key performance indicators (KPIs) to an anticipative approach based on predictable indicators to take decisions (Key Decision Indicators, KDIs). It might help managers to anticipate poor results before they occur to prevent or correct them early.

OBJECTIVE

This paper aims to identify potential KDIs and to prioritize those most relevant for high complexity hospitals.

METHODS

A narrative review was performed to identify KPIs with the potential to become KDIs. Then, two surveys were conducted with EUHA hospital managers (n = 51) to assess potential KDIs according to their relevance for decision-making (Value) and their availability and effort required to be predicted (Feasibility). Potential KDIs are prioritized for testing as predictable indicators and developing in the short term if they were classified as highly Value and Feasible.

RESULTS

The narrative review identified 45 potential KDIs out of 153 indicators and 11 were prioritized. Of nine EUHA hospitals, 25 members from seven answered, prioritizing KDIs related to the emergency department (ED), hospitalisation and surgical processes (n = 8), infrastructure and resources (n = 2) and health outcomes and quality (n = 1). The highest scores in this group were for those related to ED. The results were homogeneous among the different hospitals.

CONCLUSIONS

Potential KDIs related to care processes and hospital patient flow was the most prioritized ones to test as being predictable. KDIs represent a new approach to decision-making, whose potential to be predicted could impact the planning and management of hospital resources and, therefore, healthcare quality.

Validity evidence and psychometric evaluation of a socially accountable health index for health professions schools

There is an expectation that health professions schools respond to priority societal health needs. This expectation is largely based on the underlying assumption that schools are aware of the priority needs in their communities. This paper demonstrates how open-access, pan-national health data can be used to create a reliable health index to assist schools in identifying societal needs and advance social accountability in health professions education. Using open-access data, a psychometric evaluation was conducted to examine the reliability and validity of the Canadian Health Indicators Framework (CHIF) conceptual model. A non-linear confirmatory factor analysis (CFA) on 67 health indicators, at the health-region level (n = 97) was used to assess the model fit of the hypothesized 10-factor model. Reliability analysis using McDonald's Omega were conducted, followed by Pearson's correlation coefficients. Findings from the non-linear CFA rejected the original conceptual model structure of the CHIF. Exploratory post hoc analyses were conducted using modification indices and parameter constraints to improve model fit. A final 5-factor multidimensional model demonstrated superior fit, reducing the number of indicators from 67 to 32. The 5-factors included: Health Conditions (8-indicators); Health Functions (6-indicators); Deaths (5-indicators); Non-Medical Health Determinants (7-indicators); and Community & Health System Characteristics (6-indicators). All factor loadings were statistically significant (p < 0.001) and demonstrated excellent internal consistency ( ω >0.95). Many schools struggle to identify and measure socially accountable outcomes. The process highlighted in this paper and the indices developed serve as starting points to allow schools to leverage open-access data as an initial step in identifying societal needs.

Impact of the Italian Healthcare Outcomes Program (PNE) on the Care Quality of the Poorest Performing Hospitals

One of the main aims of the Italian National Healthcare Outcomes Program (Programma Nazionale Esiti, PNE) is the identification of the hospitals with the lowest performance, leading them to improve their quality. In order to evaluate PNE impact for a subset of outcome indicators, we evaluated whether the performance of the hospitals with the lowest scores in 2016 had significantly improved after five years. The eight indicators measured the risk-adjusted likelihood of the death of each patient (adjusted relative risk-RR) 30 days after the admission for acute myocardial infarction, congestive heart failure, stroke, chronic obstructive pulmonary disease, chronic kidney disease, femur fracture or lung and colon cancer. In 2016, the PNE identified 288 hospitals with a very low performance in at least one of the selected indicators. Overall, 51.0% (n = 147) of these hospitals showed some degree of improvement in 2021, and 27.4% of them improved so much that the death risk of their patients fell below the national mean value. In 34.7% of the hospitals, however, the patients still carried a mean risk of death >30% higher than the average Italian patient with the same disease. Only 38.5% of the hospitals in Southern Italy improved the scores of the selected indicators, versus 68.0% in Northern and Central Italy. Multivariate analyses, adjusting for the baseline performance in 2016, confirmed univariate results and showed a significantly lower likelihood of improvement with increasing hospital volume. Despite the overall methodological validity of the PNE system, current Italian policies and actions aimed at translating hospital quality scores into effective organizational changes need to be reinforced with a special focus on larger southern regions.

Measuring health equity in the ASEAN region: conceptual framework and assessment of data availability

BACKGROUND

Existing research on health equity falls short of identifying a comprehensive set of indicators for measurement across health systems. Health systems in the ASEAN region, in particular, lack a standardised framework to assess health equity. This paper proposes a comprehensive framework to measure health equity in the ASEAN region and highlights current gaps in data availability according to its indicator components.

METHODS

A comprehensive literature review was undertaken to map out a core set of indicators to evaluate health equity at the health system level. Secondary data collection was subsequently conducted to assess current data availability for ASEAN states in key global health databases, national health accounts, and policy documents.

RESULTS

A robust framework to measure health equity was developed comprising 195 indicators across Health System Inputs and Processes, Outputs, Outcomes, and Contextual Factors. Total indicator data availability equated to 72.9% (1423/1950). Across the ASEAN region, the Inputs and Processes sub-component of Health Financing had complete data availability for all indicators (160/160, 100%), while Access to Essential Medicine had the least data available (6/30, 20%). Under Outputs and Outcomes, Coverage of Selected Interventions (161/270, 59.63%) and Population Health (350/350, 100%) respectively had the most data available, while other indicator sub-components had little to none (≤ 38%). 72.145% (384/530) of data is available for all Contextual Factors. Out of the 10 ASEAN countries, the Philippines had the highest data availability overall at 77.44% (151/195), while Brunei Darussalam and Vietnam had the lowest data availability at 67.18% (131/195).

CONCLUSIONS

The data availability gaps highlighted in this study underscore the need for a standardised framework to guide data collection and benchmarking of health equity in ASEAN. There is a need to prioritise regular data collection for overlooked indicator areas and in countries with low levels of data availability. The application of this indicator framework and resulting data availability analysis could be conducted beyond ASEAN to enable cross-regional benchmarking of health equity.

Analysis of Centralized Efficiency Improvement Practices in Australian Public Health Systems

Aim

Analysis of centralized efficiency improvement practices in Australian public health systems.

Introduction

Public health systems seek to maximize outcomes generated for resources used through efficiency improvement (EI) in response to funding and demand pressures. Despite this focus, evidence for EI approaches at the whole-of-system level is lacking in the literature. There is an urgent need for evidence-based approaches to centralized EI to address these pressures. This study aims to address this gap by answering the research question "How is EI conceptualized and managed by central public health system management entities in Australia?".

Material and Methods

Document analysis was selected due to its suitability for systematically searching and appraising health system documentation, with this study following Altheide's approach focusing on whole-of-system strategic plan and management framework documents originating from Australian public health organizations.

Results

Conceptualization of efficiency varied substantially with no consistent definition identified, however common attributes included resource use, management, service and delivery. Forty-two of 43 documents contained approaches associated with improving efficiency at the whole of system level.

Discussion

While no comprehensive framework for centralized EI was evident, we identified nine core approaches which together characterize centralized EI. Together these approaches represent a comprehensive evidence-based approach to EI at the whole of system level.

Conclusion

The approaches to whole-of-system EI identified in this study are likely to be highly transferable across health systems internationally with approaches including strategic priority setting, incentivization, performance support, use of EI evidence, digital enablement and workforce capability development.

Health Management: Evaluating the Relationship between Organizational Factors, Psychosocial Risks at Work, Performance Management, and Hospital Outcomes

Introduction-Health system (HS) health organizations are complex and are in constant dynamic interaction with multiple elements, including political, environmental, societal, legal, and organizational factors, along with human components, such as human resources, patients, and other stakeholders. Objective-This research aimed to study three HS organizations, identifying and characterizing the elements of health organizations and the factors related to professionals, determining their influence on economic and financial performance results, as well as results related to the professionals and to the patients comprising the institutions. Method-A quantitative study was conducted in which data were collected through questionnaires from various sources to better understand and characterize the factors related to organizations, professionals, and patients (470 health professionals and 768 patients). To test the integrated evaluation model for health organizations, path analysis was used. Results-The results reveal that the organizational culture (OC) presents a positive relationship between the professional's quality of life (QL) and the performance management (PM) of the professionals, along with a negative relationship with the psychosocial work risks (PWR). There is also a relationship between the OC and patient satisfaction (PS), professional job satisfaction (PJS), and economic and financial results (EFR). In the relationship between the processes and the results, there are significant relationships between PM and PJS and PWR and PJS. In terms of the results, there is a significant relationship between the EFR and the PS. Conclusions-This study contributes to a deeper knowledge of the factors that influence the quality of health organizations and their results and produces recommendations for health organizations to address the current challenges.

Health System Performance and Resilience in Times of Crisis: An Adapted Conceptual Framework

With the COVID-19 pandemic, the notion of health system (HS) performance has been discussed, and the notion of resilience has become increasingly important. Lacking a recognised framework that measures the performance of HSs throughout a crisis, i.e., one that explicitly includes time as a key aspect, we examined the literature about conceptual frameworks for measuring the performance and the resilience of HSs. This review highlighted a significant diversity among 18 distinct HS performance frameworks and 13 distinct HS resilience frameworks. On this basis, we developed a model that integrates the WHO's widely recognised six building block framework in a novel approach derived from the European Observatory on HSs and Policies. The resulting framework adapts the building blocks to the different stages of a crisis, thereby allowing for a comprehensive assessment of an entire health system's performance throughout the crisis's duration, while also considering the key aspect of resilience. For a more pragmatic use of this framework in the future, indicators will be developed as a next step.

Measuring National Immunization System Performance: A Systematic Assessment of Available Resources

BACKGROUND

Vaccination coverage is widely used to assess immunization performance but, on its own, provides insufficient information to drive improvements. Assessing the performance of underlying components of immunization systems is less clear, with several monitoring and evaluation (M&E) resources available for use in different operational settings and for different purposes. We studied these resources to understand how immunization system performance is measured.

METHODS

We reviewed peer-reviewed and gray literature published since 2000 to identify M&E resources that include national-level indicators measuring the performance of immunization systems or their components (governance, financing, regulation, information systems, vaccine logistics, workforce, service delivery, and demand generation). We summarize indicators by the system components or outcomes measured and describe findings narratively.

RESULTS

We identified 20 resources to monitor immunization program objectives and guide national strategic decision-making, encompassing 631 distinct indicators. Indicators for immunization program outcomes comprised the majority (124/631 [19.7%]), largely vaccination coverage (110/124 [88.7%]). Almost all resources (19/20 [95%]) included indicators for vaccine logistics (83/631 [13.2%]), and those for regulation (19/631 [3.0%]) and demand generation (28/631 [4.4%]) were least common. There was heterogeneity in how information systems (92/563 [14.6%]) and workforce (47/631 [7.4%]) were assessed across resources. Indicators for vaccination coverage in adults, data use in decision-making, equity and diversity, effectiveness of safety surveillance, and availability of a public health workforce were notably lacking.

CONCLUSIONS

Between the resources identified in this review, we identified considerable variability and gaps in indicators assessing the performance of some immunization system components. Given the multitude of indicators, policymakers may be better served by tailoring evaluation resources to their specific context to gain useful insight into health system performance and improve data use in decision-making for immunization programs.

Analysis of Medico-Legal Complaint Data: A Retrospective Study of Three Large Italian University Hospitals

(1) Background: Identifying hospital-related critical, and excellent, areas represents the main goal of this paper, in both a national and local setting. Information was collected and organized for an internal company's reports, regarding civil litigation that has been affecting the hospital, to relate the obtained results with the phenomenon of medical malpractice on a national scale. This is for the development of targeted improvement strategies, and for investing available resources in a proficient way. (2) Methods: In the present study, data from claims management in Umberto I General Hospital, Agostino Gemelli University Hospital Foundation and Campus Bio-Medico University Hospital Foundation, from 2013 to 2020 were collected. A total of 2098 files were examined, and a set of 13 outcome indicators in the assessment of "quality of care" was proposed. (3) Results: From the total number, only 779 records (37.1%) were attributable to the categories indexable for the present analysis. This data highlights how, following a correct and rigorous categorization of hospital events, it is possible to analyze these medico-legal aspects using a small number of indicators. Furthermore, it is important to consider how a consistent percentage of remaining events was difficult to index, and was also of poor scientific interest. (4) Conclusions: The proposed indicators do not require standards to be compared to, but provide a useful instrument for comparative purposes. In fact, in addition to comparative assessment between different business realities distributed throughout the territory, the use of outcome indicators allows for a longitudinal analysis evaluating the performance of an individual structure over time.

Immigrants experience oral health care inequity: findings from Australia's National Study of Adult Oral Health

BACKGROUND

Oral health service utilization contributes to positive oral health and indicates realised access to services. The study aimed to describe patterns of oral health service use among overseas-born and Australian-born populations and assess equity in access to services.

METHODS

The study used data from Australia's National Study of Adult Oral Health 2017-2018 and was guided by the Aday and Andersen framework of access to health and Australia's National Oral Health Plan. Descriptive analyses of service use by perceived need, enabling and predisposing factors were compared between four groups: Australian-born and overseas-born who mainly speak English and Australian-born and overseas-born who mainly speak a language other than English.

RESULTS

Overseas-born who mainly speak a language other than English experienced greater oral health care inequity, largely driven by financial difficulty (avoided care due to cost: 42% vs 27%-28%; avoided/delayed visiting due to cost: 48% vs. 37%-38%; cost prevented treatment: 32% vs. 18%-24%). The most favourable visiting patterns were among the Australian-born population who speak a language other than English.

CONCLUSIONS

The study shows clear inequity experienced among immigrants in accessibility as measured through indicators of oral health care utilization and factors related to inequity, such as the ability to pay for services.

Should we adjust health expenditure for age structure on health systems efficiency? A worldwide analysis

INTRODUCTION

Healthcare expenditure, a common input used in health systems efficiency analyses is affected by population age structure. However, while age structure is usually considered to adjust health system outputs, health expenditure and other inputs are seldom adjusted. We propose methods for adjusting Health Expenditure per Capita (HEpC) for population age structure on health system efficiency analyses and assess the goodness-of-fit, correlation, reliability and disagreement of different approaches.

METHODS

We performed a worldwide (188 countries) cross-sectional study of efficiency in 2015, using a stochastic frontier analysis. As single outputs, healthy life expectancy (HALE) at birth and at 65 years-old were considered in different models. We developed five models using as inputs: (1) HEpC (unadjusted); (2) age-adjusted HEpC; (3) HEpC and the proportion of 0-14, 15-64 and 65 + years-old; (4) HEpC and 5-year age-groups; and (5) HEpC ageing index. Akaike and Bayesian information criteria, Spearman's rank correlation, intraclass correlation coefficient and information-based measure of disagreement were computed.

RESULTS

Models 1 and 2 showed the highest correlation (0.981 and 0.986 for HALE at birth and HALE at 65 years-old, respectively) and reliability (0.986 and 0.988) and the lowest disagreement (0.011 and 0.014). Model 2, with age-adjusted HEpC, presented the lowest information criteria values.

CONCLUSIONS

Despite different models showing good correlation and reliability and low disagreement, there was important variability when age structure is considered that cannot be disregarded. The age-adjusted HE model provided the best goodness-of-fit and was the closest option to the current standard.

Development of national framework for health status and health system performance indicators in Saudi Arabia

BACKGROUND

The global shift to value-based care has highlighted the necessity for performance measurement in healthcare. While the implementation may be incomplete, governments and providers have been moving towards value-based models of care delivery, funding and performance measurement.

OBJECTIVES

The purpose of this report is to illustrate the steps taken in designing the framework in Kingdom of Saudi Arabia (KSA), using a 4-step approach: Key Input Analysis, Hierarchy Design, Indicator Analysis and Selection, and Indicator Profile Development. Furthermore, it presents a comprehensive view of the indicators collected to measure performance at the level of the Health System and Health Status in KSA and highlights main patient concerns, thus providing an overview of a new, unified national framework.

METHODS

The methodology entailed examined the following components' remits and roles, reporting mechanisms and channels, and performance indicators across all pertinent National Health agencies (NHA), by conducting a SWOT analysis of each of the components across the ecosystem. Engagement with the Steering Committee members was achieved through research and interviews.

RESULTS

A total of 109 indicators were identified, of which 51 were Health Status Indicator Profiles and 58 Health System Indicators. The indicator profiles were developed with consideration of the KSA context in terms of the healthcare ecosystem as it stands today. The findings of this report, alongside the best practices arising from the benchmarking, will be key inputs into the design of the National Framework.

CONCLUSION

The regulatory entity has set out to establish a National Framework, which aims to unify performance measurement in KSA from both health system and public health perspectives across all sectors providing healthcare services, based on a list of prioritized KPIs and their complete profiles, outlining the formulae, key responsibilities, and reporting mechanisms pertaining to each one.

Quality and Performance Measurement in Primary Diabetes Care: A Qualitative Study in Urban China

BACKGROUND

Quality measurements in primary healthcare (PHC) have become an essential component for improving diabetes outcomes in many high-income countries. However, little is known about their implementation within the Chinese health-system context and how they are perceived by patients, physicians, and policy-makers. We examined stakeholders' perceptions of quality and performance measurements for primary diabetes care in Shanghai, China, and analyzed facilitators and barriers to implementation.

METHODS

In-depth interviews with 26 key stakeholders were conducted from 2018 to 2019. Participants were sampled from two hospitals, four community healthcare centers (CHCs), and four institutes involved in regulating CHCs. The Consolidated Framework for Implementation Research (CFIR) guided data analysis.

RESULTS

Existing quality measurements were uniformly implemented via a top-down process, with daily monitoring of family doctors' work and pay-for-performance incentives. Barriers included excluding frontline clinicians from indicator planning, a lack of transparent reporting, and a rigid organizational culture with limited bottom-up feedback. Findings under the CFIR construct "organizational incentives" suggested that current pay-for-performance incentives function as a "double-edged sword," increasing family doctors' motivation to excel while creating pressures to "game the system" among some physicians. When considering the CFIR construct "reflecting and evaluating," policy-makers perceived the online evaluation application - which provides daily reports on family doctors' work - to be an essential tool for improving quality; however, this information was not visible to patients. Findings included under the "network and communication" construct showed that specialists support the work of family doctors by providing training and patient consultations in CHCs.

CONCLUSION

The quality of healthcare could be considerably enhanced by involving patients and physicians in decisions on quality measurement. Strengthening hospital-community partnerships can improve the quality of primary care in hospital-centric systems. The case of Shanghai provides compelling policy lessons for other health systems faced with the challenge of improving PHC.

Common Features of Selection Processes of Health System Performance Indicators in Primary Healthcare: A Systematic Review

BACKGROUND

Health system performance indicators are widely used to assess primary healthcare (PHC) performance. Despite the numerous tools and some convergence on indicator criteria, there is not a clear understanding of the common features of indicator selection processes. We aimed to review the literature to identify papers that document indicator selection processes for health system performance indicators in PHC.

METHODS

We searched the online databases Scopus, Medline, and CINAHL, as well as the grey literature, without time restrictions, initially on July 31, 2019 followed by an update November 13, 2020. Empirical studies or reports were included if they described the selection of health system performance indicators or frameworks, that included PHC indicators. A combination of the process focussed research question and qualitative analysis meant a quality appraisal tool or assessment of bias could not meaningfully be applied to assess individual studies. We undertook an inductive analysis based on potential indicator selection processes criteria, drawn from health system performance indicator appraisal tools reported in the literature.

RESULTS

We identified 16 503 records of which 28 were included in the review. Most studies used a descriptive case study design. We found no consistent variations between indicator selection processes of health systems of high income and low- or lower-middle income countries. Identified common features of selection processes for indicators in PHC include literature review or adaption of an existing framework as an initial step; a consensus building process with stakeholders; structuring indicators into categories; and indicator criteria focusing on validity and feasibility. The evidence around field testing with utility and consideration of reporting burden was less clear.

CONCLUSION

Our findings highlight several characteristics of health system indicator selection processes. These features provide the groundwork to better understand how to value indicator selection processes in PHC.

The value of vaccines

Optimizing vaccine spending depends on recognizing the full value of vaccination (VoV). Existing taxonomies of such value are not comprehensive because they are not guided by general theories. I rely on two such theories: subjective-value theory claims that what has value is determined by what people actually or ideally want in life. A welfarist theory of government states that a fundamental objective of government is to promote social value (or social welfare). These jointly imply that any aspect of life that individuals actually or ideally value and that could be negatively affected by vaccine-preventable diseases (and therefore positively affected by preventive vaccines) is an element of VoV. I build a more comprehensive-value taxonomy than currently exists based on this implication.

Validation and endorsement of health system performance measures for opioid use disorder in British Columbia, Canada: A Delphi panel study

Background

Limited data exists on the performance of the healthcare system in opioid use disorder (OUD). We evaluated the face validity and potential risks of a set of health system performance measures for OUD collaboratively with clinicians, policymakers and people with lived experience of opioid use (PWLE) in the interest of establishing an endorsed set of measures for public reporting.

Methods

Through a two-stage Delphi-panel approach, a panel of clinical and policy experts validated and considered 102 previously constructed OUD performance measures for endorsement using information on measurement construction, sensitivity analyses, quality of evidence, predictive validity, and feedback from local PWLE. We collected quantitative and qualitative survey responses from 49 clinicians and policymakers, and 11 PWLE. We conducted inductive and deductive thematic analysis to present qualitative responses.

Results

A total of 37 measures of 102 were strongly endorsed (9/13 cascade of care, 2/27 clinical guideline compliance, 17/44 healthcare integration, and 9/18 healthcare utilization measures). Thematic analysis of responses revealed several themes regarding measurement validity, unintended consequences, and key contextual considerations. Overall, measures related to the cascade of care (excluding opioid agonist treatment dose tapering) received strong endorsements. PWLE highlighted barriers to accessing treatment, undignified aspects of treatment, and lack of a full continuum of care as their concerns.

Conclusion

We defined 37 endorsed health system performance measures for OUD and presented a range of perspectives on their validity and use. These measures provide critical considerations for health system improvement in the care of people with OUD.

Identifying requisite learning health system competencies: a scoping review

OBJECTIVES

Learning health systems (LHS) integrate knowledge and practice through cycles of continuous quality improvement and learning to increase healthcare quality. LHS have been conceptualised through multiple frameworks and models. Our aim is to identify and describe the requisite individual competencies (knowledge, skills and attitudes) and system competencies (capacities, characteristics and capabilities) described in existing literature in relation to operationalising LHS.

METHODS

A scoping review was conducted with descriptive and thematic analysis to identify and map competencies of LHS for individuals/patients, health system workers and systems. Articles until April 2020 were included based on a systematic literature search and selection process. Themes were developed using a consensus process until agreement was reached among team members.

RESULTS

Eighty-nine articles were included with most studies conducted in the USA (68 articles). The largest number of publications represented competencies at the system level, followed by health system worker competencies. Themes identified at the individual/patient level were knowledge and skills to understand and share information with an established system and the ability to interact with the technology used to collect data. Themes at the health system worker level were skills in evidence-based practice, leadership and teamwork skills, analytical and technological skills required to use a 'digital ecosystem', data-science knowledge and skill and self-reflective capacity. Researchers embedded within LHS require a specific set of competencies. Themes identified at the system level were data, infrastructure and standardisation; integration of data and workflow; and culture and climate supporting ongoing learning.

CONCLUSION

The identified individual stakeholder competencies within LHS and the system capabilities of LHS provide a solid base for the further development and evaluation of LHS. International collaboration for stimulating LHS will assist in further establishing the knowledge base for LHS.

Healthcare Service Interventions to Improve the Healthcare Outcomes of Hospitalised Patients with Extreme Obesity: Protocol for an Evidence and Gap Map

Hospitalised patients with extreme obesity have poorer healthcare outcomes compared to normal weight patients. How hospital services are coordinated and delivered to meet the care needs of patients with extreme obesity is not well understood. The aim of the proposed evidence gap map (EGM) is to identify and assess the available evidence on healthcare interventions to improve healthcare outcomes for hospitalised patients with extreme obesity. This research will use standardised evidence gap map methods to undertake a five-stage process to develop an intervention–outcome framework; identify the current evidence; critically appraise the quality of the evidence, extract, code, and summarise the data in relation to the EGM objectives; and create a visualisation map to present findings. This EGM will provide a means of determining the nature and quality of health service initiatives available, identify the components of the services delivered and the outcome measures used for evaluation, and will identify areas where there is a lack of research that validates the funding of new research studies.

Development and validation of health system performance measures for opioid use disorder in British Columbia, Canada

BACKGROUND

Performance measurement provides an evidence-based means to inform development of interventions to improve the quality of care for people who use opioids. We aimed to develop and assess the predictive validity of health system performance measures for opioid use disorder (OUD) in British Columbia (BC), Canada.

METHODS

Performance measures were generated using retrospective population-level administrative datasets (both provincial and regional) and publicly-reported retrospective data according to four domains (care engagement, clinical guideline compliance, integration, and healthcare utilization). The adjusted odds ratio was estimated via generalized linear mixed models to determine predictive validity for all-cause hospitalization or mortality within 6 months of measurement.

FINDINGS

A total of 102 performance measures were constructed. We identified 55,470 diagnosed PWOUD, and 39,456 ever engaged in opioid agonist treatment (OAT). We found divergent rates of treatment for concurrent conditions (7.4% for alcohol use disorder to 80.1% for HIV/AIDS), low levels of linkage to OAT and other outpatient care following acute care, and increasing levels of service provision, including increases in OAT prescribers and pharmacies, naloxone kit distribution and overdose prevention site visitation. Our analyses on the predictive validity measures largely supported a priori hypotheses on the direction of effect on the outcome.

CONCLUSIONS

We identified a range of priorities to improve the quality of care for PWOUD, with critical gaps in linkage to care through acute care settings and long-term engagement in OAT. The proposed measures can be derived for geographic and clinical subgroups and updated over time, providing a basis to monitor and evaluate efforts to address the public health burden of OUD.

The evolution and inter-sectoral interaction mechanism of China's national medical alliance: An analysis based on complex systems theory

This work investigates the performance and inter-sectoral interaction mechanism of China's largest vertically integrated care network, the national medical alliance (NMA). The data collected derive from the China Health Statistics Bulletin and the China Health Statistical Yearbook for the period 2009-2018. The data include 64 observation indicators for five medical sectors in the NMA, namely, tertiary hospitals (THS), secondary hospitals (SHS), community health centres (CHCS), township hospitals (TsHS) and professional public health institutions (PPHIS). This research combines complex systems theory with a multilevel structural dynamic factor model, and yields two main results. First, although the trend for the NMA's global factor is increasing, the evolutionary paths for sectoral factors differ substantially. Among the sectoral factors, the sectoral factor of THS continued to decline, and neither the sectoral factor of CHCS nor the sectoral factor of TsHS has significantly improved. Then, the interaction mechanism between the various NMA sectors is investigated. While a close relationship has been formed between THS and CHCS and between SHS and CHCS, there remains no close two-way relationship between either THS and TsHS or THS and SHS. Thus, going forward, to reach the policy expectations, China's NMA implementation must consider the interaction between different constituent sectors.

Does the mental health system provide effective coverage to people with schizophrenic disorder? A self-controlled case series study in Italy

PURPOSE

To measure indicators of timeliness and continuity of treatments on patients with schizophrenic disorder in 'real-life' practice, and to validate them through their relationship with relapse occurrences.

METHODS

The target population was from four Italian regions overall covering 22 million beneficiaries of the NHS (37% of the entire Italian population). The cohort included 12,054 patients newly taken into care for schizophrenic disorder between January 2015 and June 2016. The self-controlled case series (SCCS) design was used to estimate the incidence rate ratio of relapse occurrences according to mental healthcare coverage.

RESULTS

Poor timeliness (82% and 33% of cohort members had not yet started treatment with psychosocial interventions and antipsychotic drug therapy within the first year after they were taken into care) and continuity (27% and 23% of patients were persistent with psychosocial interventions, and antipsychotic drug therapy within the first 2 years after starting the specific treatment) were observed. According to SCCS design, 4794 relapses occurred during 9430 PY (with incidence rate of 50.8 every 100 PY). Compared with periods not covered by mental healthcare, those covered by psychosocial intervention alone, antipsychotic drugs alone and by psychosocial intervention and antipsychotic drugs together were, respectively, associated with relapse rate reductions of 28% (95% CI 4-46%), 24% (17-30%) and 44% (32-53%).

CONCLUSION

Healthcare administrative data may contribute to monitor and to assess the effectiveness of a mental health system. Persistent use of both psychosocial intervention and antipsychotic drugs reduces risk of severe relapse.

Towards a harmonised framework for developing quality of care indicators for global health: a scoping review of existing conceptual and methodological practices

OBJECTIVES

Despite significant advances in the science of quality of care measurement over the last decade, approaches to developing quality of care indicators for global health priorities are not clearly defined. We conducted a scoping review of concepts and methods used to develop quality of healthcare indicators to better inform ongoing efforts towards a more harmonised approach to quality of care indicator development in global health.

METHODS

We conducted a systematic search of electronic databases, grey literature and references for articles on developing quality of care indicators for routine monitoring in all healthcare settings and populations, published in English between 2010 and 2020. We used well-established methods for article screening and selection, data extraction and management. Results were summarised using a descriptive analysis and a narrative synthesis.

RESULTS

The 221 selected articles were largely from high-income settings (89%), particularly the USA (46%), Canada (9%), UK (9%) and Europe (17%). Quality of care indicators were developed mainly for healthcare providers (56%), for benchmarking or quality assurance (37%) and quality improvement (29%), in hospitals (32%) and primary care (26%), across many diseases. The terms 'quality indicator' and 'quality measure' were the most frequently encountered terms (50% and 21%, respectively). Systematic approaches for quality of care indicator development emerged within national quality of care systems or through cross-country collaborations in high-income settings. Maternal, neonatal and child health (33%), mental health (26%) and primary care (57%) studies applied most components of systematic approaches, but not consistently or rigorously.

DISCUSSION

The current evidence shows variations in concepts and approaches to developing quality of care indicators, with development and application mainly in high-income countries.

CONCLUSION

Additional efforts are needed to propose 'best-practice' conceptual frameworks and methods for developing quality of care indicators to improve their utility in global health measurement.

Engaging citizens in the development of a health system performance assessment framework: a case study in Ireland

Background

The launch in 2017 of the Irish 10-year reform programme Sláintecare represents a key commitment in the future of the health system. An important component of the programme was the development of a health system performance assessment (HSPA) framework. In 2019, the Department of Health of Ireland (DoH) and Health Service Executive (HSE) commissioned the technical support of researchers to develop an outcome-oriented HSPA framework which should reflect the shared priorities of multiple stakeholders, including citizens. This study describes the method applied in the Irish context and reflects on the added value of using a citizen panel in the development of an HSPA framework.

Methods

A panel of 15 citizens was convened, recruited by a third-party company using a sampling strategy to achieve a balanced mix representing the Irish society. Panellists received lay-language preparatory materials before the meeting. Panellists used a three-colour scheme to signal the importance of performance measures. An exit questionnaire was administered to understand how participants experienced being part of the panel. The citizen panel was the first in a series of three panels towards the development of the HSPA framework, followed by panels including representatives of the DoH and HSE, and representatives from professional associations and special interest groups.

Results

The citizen panel generated 249 health performance measures ranging across 13 domains. Top-ranking domains to the citizen panel (people-centredness, coordination of care, and coverage) were less prioritized by the other panels; domains less prioritized by the citizen panel, such as accessibility, responsiveness, efficiency, and effectiveness, were of higher priority in the other panels. Citizen panellists shared a similar understanding of what a citizen panel involves and described their experience at the panel as enjoyable, interesting, and informative.

Conclusions

The priorities of the citizen panel were accounted for during all phases of developing the HSPA framework. This was possible by adopting an inclusive development process and by engaging citizens early on. Citizen engagement in HSPA development is essential for realizing citizen-driven healthcare system performance and generating trust and ownership in performance intelligence. Future research could expand the use of citizen panels to assess, monitor, and report on the performance of healthcare systems.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00798-8.

Exploring the actionability of healthcare performance indicators for quality of care: a qualitative analysis of the literature, expert opinion and user experience

BACKGROUND

This study explores the meaning of actionable healthcare performance indicators for quality of care-related decisions. To do so, we analyse the constructs of fitness for purpose and fitness for use across healthcare systems and in practice based on the literature, expert opinion and user experience.

METHODS

A multiphase qualitative study was undertaken. Phases included a literature review, a first round of one-on-one interviews with a panel of academics and thought leaders in the field (n=16), and a second round of interviews with real-world users of performance indicators (n=16). Thematic analysis was conducted between phases in order to triangulate findings in a stepwise process.

RESULTS

Common uses of healthcare performance indicators were differentiated within micro-meso-macro contexts of healthcare systems. Each purpose of use signals different decision-making tasks, and in effect information needs. An indicator's fitness for use can be appraised by three clusters of considerations: methodological, contextual and managerial. Methodological considerations gauge an indicator's perceived importance, engagement potential, interpretability, standardisation, feasibility of remedial actions, alignment to care models and sensitivity to change. Information infrastructure, system governance, workforce capacity and learning culture were found as enabling contextual considerations. Managerial considerations influencing an indicator's use in practice were found to span the selection of indicators, data collection, analysis, display of results and delivery of information to decision-makers.

CONCLUSION

The actionability of a healthcare performance indicator should be appraised by its alignment with the intended purpose of use beyond aggregate healthcare system levels, in combination with the extent to which methodological, contextual and managerial fitness for use considerations are met. Striking a better balance between the importance weighted to an indicator's statistical merits and emphasis put to its fitness for purpose and use is needed for indicators that are ultimately actionable for quality of care-related decision-making.

Variability in the assessment of children's primary healthcare in 30 European countries

Background

The high variability in the types and number of measures adopted to evaluate childcare across European countries makes it necessary to investigate country practices to identify trends in setting national priorities in the assessment of child well-being.

Objective

This paper intends to investigate country practices under the lens of variability to explore possible trends in setting national priority in the evaluation of childcare. In particular, it analyses variability considering to what extent this depends on the tendency of adopting a broad vision (i.e. selecting measures for a larger variety of aspects) or whether this is influenced by the choice of adopting an in-depth approach (i.e. using more measures to analyse a specific aspect)

Methods

An ad hoc questionnaire was administered to a national expert in each country and yielded 352 measures. To analyse variability, the breadth in the number of aspects considered was explored using a convergence index, while the depth in the distribution of measures in each aspect was investigated by computing a coefficient of variation. Countries were grouped by adopting a hierarchical clustering approach.

Results

There is a high variability across countries in the selection of measures that cover different aspects of childcare. Preferences in the distribution of measures are significant even at the domain level and in countries that use a limited number of measures and become more evident at the category and sub-category levels. The statistical analysis clusters countries in four main groups and two outliers. The in-depth distribution of measures focused on a specific aspect shows a homogeneous pattern, with the identification of two main groups of countries.

Conclusions

A limited set of measures are shared across countries hampering a robust comparison of paediatric models. The selection of measures shows that the evaluation is closely related to national priorities as resulting from the number and types of measures adopted. Moreover, a range of a reasonable number of measures can be hypothesized to address the quality of childcare under a multi-dimensional perspective.

Stakeholder Perceptions of Key Aspects of High-Quality Cancer Care to Assess with Patient Reported Outcome Measures: A Systematic Review

Simple Summary

We conducted a review to identify important symptoms reported by patients on questionnaires (e.g., pain) that can be used to compare cancer centers on how well they provide care. For example, cancer centers could be compared on the percentage of patients with controlled pain after adjusting for demographic and clinical characteristics. Standard review methods were used to identify studies through August 2020. Searches generated 1813 articles and 1779 were coded as not relevant. The remaining 34 studies showed that patients, caregivers, clinicians, and healthcare administrators identify psychosocial care (e.g., distress) and symptom management as critical parts of high-quality care. Patients and caregivers also perceive that maintaining physical function and daily activities are important. Clinicians and healthcare administrators perceive control of specific symptoms to be important (e.g., pain, poor sleep, diarrhea). Results were used to inform testing of symptom questionnaires to compare the quality of care provided by six cancer centers.

Abstract

Performance measurement is the process of collecting, analyzing, and reporting standardized measures of clinical performance that can be compared across practices to evaluate how well care was provided. We conducted a systematic review to identify stakeholder perceptions of key symptoms and health domains to test as patient-reported performance measures in oncology. Stakeholders included cancer patients, caregivers, clinicians, and healthcare administrators. Standard review methodology was used, consistent with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). MEDLINE/PubMed, EMBASE, and the Cochrane Library were searched to identify relevant studies through August 2020. Four coders independently reviewed entries and conflicts were resolved by a fifth coder. Efficacy and effectiveness studies, and studies focused exclusively on patient experiences of care (e.g., communication skills of providers) were excluded. Searches generated 1813 articles and 1779 were coded as not relevant, leaving 34 international articles for extraction. Patients, caregivers, clinicians, and healthcare administrators prioritize psychosocial care (e.g., distress) and symptom management for patient-reported performance measures. Patients and caregivers also perceive that maintaining physical function and daily activities are critical. Clinicians and administrators perceive control of specific symptoms to be critical (gastrointestinal symptoms, pain, poor sleep). Results were used to inform testing at six US cancer centers.

Performance management in complex adaptive systems: a conceptual framework for health systems

Existing performance management approaches in health systems in low-income and middle-income countries are generally ineffective at driving organisational-level and population-level outcomes. They are largely directive: they try to control behaviour using targets, performance monitoring, incentives and answerability to hierarchies. In contrast, enabling approaches aim to leverage intrinsic motivation, foster collective responsibility, and empower teams to self-organise and use data for shared sensemaking and decision-making.The current evidence base is too limited to guide reforms to strengthen performance management in a particular context. Further, existing conceptual frameworks are undertheorised and do not consider the complexity of dynamic, multilevel health systems. As a result, they are not able to guide reforms, particularly on the contextually appropriate balance between directive and enabling approaches. This paper presents a framework that attempts to situate performance management within complex adaptive systems. Building on theoretical and empirical literature across disciplines, it identifies interdependencies between organisational performance management, organisational culture and software, system-level performance management, and the system-derived enabling environment. It uses these interdependencies to identify when more directive or enabling approaches may be more appropriate. The framework is intended to help those working to strengthen performance management to achieve greater effectiveness in organisational and system performance. The paper provides insights from the literature and examples of pitfalls and successes to aid this thinking. The complexity of the framework and the interdependencies it describes reinforce that there is no one-size-fits-all blueprint for performance management, and interventions must be carefully calibrated to the health system context.

Developing clinical indicators for oncology: the inaugural cancer care indicator set for the Australian Council on Healthcare Standards

INTRODUCTION

The Australian Council on Healthcare Standards (ACHS) sponsored an expert-led, consensus-driven, four-stage process, based on a modified Delphi methodology, to determine a set of clinical indicators as quality measures of cancer service provision in Australia. This was done in response to requests from institutional health care providers seeking accreditation, which were additional and complementary to the existing radiation oncology set. The steering group members comprised multidisciplinary key opinion leaders and a consumer representative. Five additional participants constituted the stakeholder group, who deliberated on the final indicator set.

METHODS AND RECOMMENDATIONS

An initial meeting of the steering group scoped the high level nature of the desired set. In stage 2, 65 candidate indicators were identified by a literature review and a search of international metrics. These were ranked by survey, based on ease of data accessibility and collectability and clinical relevance. The top 27 candidates were debated by the stakeholder group and culled to a final set of 16 indicators. A user manual was created with indicators mapped to clinical codes. The indicator set was ratified by the Clinical Oncology Society of Australia and is now available for use by health care organisations participating in the ACHS Clinical Indicator Program. This inaugural cancer clinical indicator set covers high level assessment of various critical processes in cancer service provision in Australia. Regular reviews and updates will ensure usability.

CHANGES IN MANAGEMENT AS A RESULT OF THIS STATEMENT

This is the inaugural indicator set for cancer care for use across Australia and internationally under the ACHS Clinical Indicator Program. Multidisciplinary involvement through a modified Delphi process selected indicators representing both generic and specific aspects of care across the cancer journey pathway and will provide a functional tool to compare health care delivery across multiple settings. It is anticipated that this will drive continual improvement in cancer care provision.

Healthcare planning across healthcare sectors in Baden-Wuerttemberg, Germany: a stakeholder online survey to identify indicators

Background

Stakeholders in the German state of Baden-Wuerttemberg agreed upon the central aims for healthcare planning. These include a focus on geographical districts; a comprehensive, cross-sectoral perspective on healthcare needs and services; and use of regional data for healthcare planning. Therefore, healthcare data at district level is needed. Nevertheless, decision makers face the challenge to make a selection from numerous indicators and frameworks, which all have limitations or do not well apply to the targeted setting. The aim of this study was to identify district level indicators to be used in Baden-Wuerttemberg for the purpose of cross-sectoral and needs-based healthcare planning involving stakeholders of the health system.

Methods

A conceptual framework for indicators was developed. A structured search for indicators identified 374 potential indicators in indicator sets of German and international institutions and agencies (n = 211), clinical practice guidelines (n = 50), data bases (n = 35), indicator databases (n = 25), published literature (n = 35), and other sources (n = 18). These indicators were categorised according to the developed framework dimensions. In an online survey, institutions of various stakeholders were invited to assess the relevance of these indicators from December 2016 until January 2017. Indicators were selected in terms of a median value of the assessed relevance.

Results

22 institutions selected 212 indicators for the five dimensions non-medical determinants of health (20 indicators), health status (25), utilisation of the health system (34), health system performance (87), and healthcare provision (46).

Conclusions

Stakeholders assessed a large number of indicators as relevant for use in healthcare planning on district level.

Trial registration

Not applicable.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-06514-0.

A Novel 3-Part Approach to Tackle the Problem of Health Inequities in Early Childhood

The first 5 years of a child's life are crucial in laying the foundation for their health and developmental trajectory into adulthood. These early years are especially influenced by the surrounding environments in which children live and grow. A large international body of evidence demonstrates that children who experience disadvantage tend to fall increasingly behind over time. At the societal level, these inequities can cause substantial social burdens and significant costs across health, education, and welfare budgets. A contributing factor is that children experiencing adversity are less likely to have access to the environmental conditions that support them to thrive. Many of these factors are modifiable at the community or place level. We argue for three key-though not exhaustive-ideas that collectively could achieve more equitable outcomes for children facing disadvantage and experiencing adversity:We conclude that if adopted, these 3 ideas could contribute to the ability of local communities and networks to identify and respond to factors that address early childhood inequalities.

Guideline-based quality assurance: a conceptual framework for the definition of key elements

BACKGROUND

In 2017, the European Commission's Joint Research Centre (JRC) started developing a methodological framework for a guideline-based quality assurance (QA) scheme to improve cancer quality of care. During the first phase of the work, inconsistency emerged about the use of terminology for the definition, the conceptual underpinnings and the way QA relates to health questions that are answered in guidelines. The objective of this final of three articles is to propose a conceptual framework for an integrated approach to guideline and QA development and clarify terms and definitions for key elements. This work will inform the upcoming European Commission Initiative on Colorectal Cancer (ECICC).

METHODS

A multidisciplinary group of 23 experts from key organizations in the fields of guideline development, performance measurement and quality assurance participated in a mixed method approach including face-to-face dialogue and several rounds of virtual meetings. Informed by results of a systematic literature review that indicated absence of an existing framework and practical examples, we first identified the relations of key elements in guideline-based QA and then developed appropriate concepts and terminology to provide guidance.

RESULTS

Our framework connects the three key concepts of quality indicators, performance measures and performance indicators integrated with guideline development. Quality indicators are constructs used as a guide to monitor, evaluate, and improve the quality of the structure, process and outcomes of healthcare services; performance measures are tools that quantify or describe measurable elements of practice performance; and performance indicators are quantifiable and measurable units or scores of practice, which should be guided by guideline recommendations.

CONCLUSIONS

The inconsistency in the way key terms of QA are used and defined has confused the field. Our conceptual framework defines the role, meaning and interactions of the key elements for improving quality in healthcare. It directly builds on the questions asked in guidelines and answered through recommendations. These findings will be applied in the forthcoming ECICC and for the future updates of ECIBC. These are large-scale integrated projects aimed at improving healthcare quality across Europe through the development of guideline-based QA schemes; this will help in implementing and improving our approach.

Comparison of Crude Population-Level Indicators of Opioid Use and Related Harm in New Zealand and Ontario (Canada)

North America and select other Commonwealth jurisdictions have been experiencing unprecedented opioid epidemics characterized by excessive and persistently high levels of opioid misuse, morbidity and mortality, and related disease burden. Recent discussions have considered whether New Zealand might undergo or needs to expect a similar ‘opioid crisis’. Towards further informing these considerations, we examine and compare essential, publicly available indicators of opioid utilization and harms (mortality) from New Zealand and the Canadian province of Ontario, due to the fact that both operate public health care systems in similar socio-cultural settings. We find that the two jurisdictions have featured vastly different population levels of opioid exposure, opioid consumption patterns (e.g., high-dose/long-term/high-risk prescribing) known as key predictors of adverse outcomes, and levels of opioid mortality as evidenced by concrete epidemiological indicators and data. Specifically for opioid-related death rates, these were already approximately threefold higher in Ontario compared to New Zealand based on most recent comparison data (e.g., 2012); these differentials have likely further grown more recently given major and distinct changes in population-level opioid exposure and risks, and subsequent opioid-related deaths since then in Ontario. Based on the present data and related evidence, New Zealand does not seem to need to anticipate an opioid mortality epidemic similar to that experienced in North America; however, it would be of interest to establish more comprehensive and timely surveillance of key system-level indicators of opioid use and harms as are standard in North America. As such, this inter-jurisdictional comparison makes for a case study in starkly contrasting scenarios of opioid use and harms, the drivers behind which deserve further systematic examination.

CMIID: A comprehensive medical information identifier for clinical search harmonization in Data Safe Havens

Over the last decades clinical research has been driven by informatics changes nourished by distinct research endeavors. Inherent to this evolution, several issues have been the focus of a variety of studies: multi-location patient data access, interoperability between terminological and classification systems and clinical practice and records harmonization. Having these problems in mind, the Data Safe Haven paradigm emerged to promote a newborn architecture, better reasoning and safe and easy access to distinct Clinical Data Repositories. This study aim is to present a novel solution for clinical search harmonization within a safe environment, making use of a hybrid coding taxonomy that enables researchers to collect information from multiple repositories based on a clinical domain query definition. Results show that is possible to query multiple repositories using a single query definition based on clinical domains and the capabilities of the Unified Medical Language System, although it leads to deterioration of the framework response times. Participants of a Focus Group and a System Usability Scale questionnaire rated the framework with a median value of 72.5, indicating the hybrid coding taxonomy could be enriched with additional metadata to further improve the refinement of the results and enable the possibility of using this system as data quality tagging mechanism.

[Introduction to the quality of medical journalism and initial assessments of COVID-19 media coverage]

Die Medienlandschaft unterliegt einem stetigen Wandel, innovative Medientechnologien beeinflussen in immer stärkerem Maße unsere Lebenswelt. Da digitale Medien zunehmend in allen Altersklassen konsumiert werden und Laien sich im Internet auch häufig zu gesundheitlichen und medizinischen Themen informieren, stellt sich die Frage nach der Qualität dieser (journalistischen) Inhalte im Web. Die Rolle von Journalistinnen und Journalisten wandelt sich und für die Wissenschaft gibt es neue Möglichkeiten und Herausforderungen bei der digitalen Verbreitung von Forschungsergebnissen. Welchen qualitativen Ansprüchen müssen diese Inhalte genügen und welche Potenziale und Chancen, aber eben auch Risiken bringen online verbreitete und rezipierte Informationen mit sich?

Im vorliegenden Beitrag werden Themen der Medizin- und Gesundheitskommunikation und Prozesse der journalistischen Berichterstattung beschrieben. Anschließend diskutieren wir das in der Medizin bereits etablierte Kriterium der „Evidenz“ als möglichen Ansatz zur Beurteilung gesundheitsrelevanter und medizinischer Informationen. Abschließend wird die COVID-19-Pandemie im Kontext des wachsenden Medienpopulismus diskutiert.

General practitioner follow-up after hospitalisation in Central and Eastern Sydney, Australia: access and impact on health services

Objectives General practitioner (GP) follow-up after a hospital admission is an important indicator of integrated care. We examined the characteristics of patients who saw a GP within 2 weeks of hospital discharge in the Central and Eastern Sydney (CES) region, Australia, and the relationship between GP follow-up and subsequent hospitalisation. Methods This data linkage study used a cohort of 10240 people from the 45 and Up Study who resided in CES and experienced an overnight hospitalisation in the 5 years following recruitment (2007-14). Characteristics of participants who saw a GP within 2 weeks of discharge were compared with those who did not using generalised linear models. Time to subsequent hospitalisation was compared for the two groups using Cox proportional hazards regression models stratified by prior frequency of GP use. Results Within 2 weeks of discharge, 64.3% participants saw a GP. Seeing a GP within 2 weeks of discharge was associated with lower rates of rehospitalisation for infrequent GP users (i.e. <8 visits in year before the index hospitalisation; hazard ratio (HR) 0.83; 95% confidence interval (CI) 0.70-0.97) but not frequent GP users (i.e. ≥8 plus visits; HR 1.02; 95% CI 0.90-1.17). Conclusion The effect of seeing a GP on subsequent hospitalisation was protective but differed depending on patient care needs. What is known about the topic? There is general consensus among healthcare providers that primary care is a significant source of ongoing health care provision. What does this paper add? This study explored the relationship between GP follow-up after an uncomplicated hospitalisation and its effect on rehospitalisation. What are the implications for practitioners? Discharge planning and the transfer of care from hospital to GP through discharge arrangements have substantial benefits for both patients and the health system.

Contracting treatment services in Australia: Do measures adhere to best practice?

BACKGROUND

Contracting non-government services to provide alcohol and other drug (AOD) treatment requires the specification of performance measures to ensure accountability for public funds. There is currently no standardised approach by funders to the measurement of performance of AOD treatment in Australia. Funding to non-government organisations (NGOs) is complex, with NGOs managing funding agreements from a range of sources that contain a large number of differing performance measures. This study aimed to assess performance measures used in contracts for NGO AOD treatment providers and how they align with best practice.

METHODS

Performance measures contained in funding agreements for treatment providers were collected from a diverse sample of both funders (n=8) and treatment providers (n=20) resulting in more than 1,100 measures. The list of measures was synthesized to a finite, non-duplicative list (n=537). Measures were assessed by three raters against 11 criteria (for example, measurable and timely) documented as best practice elements of performance measures (Council of Australian Governments, 2011a). Measures were also coded into different measurement types (such as outcome and process).

RESULTS

None of the 537 unique performance measures used in funding agreements for treatment providers fully met the criteria for best practice in performance measurement. Whilst the literature and government policy is being directed towards outcomes-based funding and reporting, only 7.6% of measures were classified as outcome measures. The majority of measures were classified at output (41.3%) and process (23.6%) measures.

CONCLUSION

Current measures in contracts applied by funders to treatment services do not adhere to best practice. The development and implementation of new performance measures is required to inform AOD policy and accountability of public funds. Further, identifying more robust performance measures has the potential to lead to reduced reporting burden on service providers and better monitoring of service quality and outcomes.

Healthcare experiences of patients with chronic heart failure in Germany: a scoping review

OBJECTIVES

To review systematically the past 10 years of research activity into the healthcare experiences (HCX) of patients with chronic heart failure (CHF) in Germany, in order to identify research foci and gaps and make recommendations for future research.

DESIGN

In this scoping review, six databases and grey literature sources were systematically searched for articles reporting HCX of patients with CHF in Germany that were published between 2008 and 2018. Extracted results were summarised using quantitative and qualitative descriptive analysis.

RESULTS

Of the 18 studies (100%) that met the inclusion criteria, most were observational studies (60%) that evaluated findings quantitatively (60%). HCX were often concerned with patient information, global satisfaction as well as relationships and communication between patients and providers and generally covered ambulatory care, hospital care and rehabilitation services. Overall, the considerable heterogeneity of the included studies' outcomes only permitted relatively trivial levels of synthesis.

CONCLUSION

In Germany, research on HCX of patients with CHF is characterised by missing, inadequate and insufficient information. Future research would benefit from qualitative analyses, evidence syntheses, longitudinal analyses that investigate HCX throughout the disease trajectory, and better reporting of sociodemographic data. Furthermore, research should include studies that are based on digital data, reports of experiences gained in under-investigated yet patient-relevant healthcare settings and include more female subjects.

Women's satisfaction with the quality of antenatal care services rendered at public health facilities in Northwest Ethiopia: the application of partial proportional odds model

OBJECTIVES

The study was aimed: (1) to describe the quality of antenatal care (ANC) at public health facilities in Northwest Ethiopia, including dimensions of the structure, process and outcome; and (2) to assess the relationship between ANC satisfaction and structure and process dimension of ANC quality.

DESIGN

Cross sectional.

SETTING

Healthcare facilities providing ANC services in Northwest Ethiopia.

PARTICIPANTS

795 pregnant women attending the antenatal clinics at 15 public health facilities and 41 health workers working for the surveyed facilities.

OUTCOME MEASURES

The outcome variable, women's satisfaction with ANC, was constructed from multiple satisfaction items using principal component analysis on an ordered, categorical and three-point Likert scale. The key hypothesised factors considered were structural and process aspects of care. Data were analysed using the partial proportional odds model with 95% CI.

RESULTS

The result revealed that only 30.3% of the pregnant women were highly satisfied, whereas 31.7% had a lower satisfaction level. The findings showed that process quality indicators better predicted client satisfaction. In relation to this, better scores in history taking (aOR₁=aOR₂; 1.81 (95% CI 1.25 to 2.60)), counselling (aOR₁ = aOR₂; 1.89 (95% CI 1.33 to 2.69)) and screening (aOR₁= aOR₂; 18.10 (95% CI 11.52 to 28.39)) were associated with achieving higher satisfaction. We also observed a significant but lower satisfaction among women in the late trimester of pregnancy (aOR₁ = aOR₂; 0.87 (95% CI 0.78 to 0.97)). However, we did not see any significant relationship between structural variables and client satisfaction.

CONCLUSIONS

The study demonstrated that women's satisfaction with ANC was low. The contents of ANC services covered during client-provider interaction were the main factors affecting client satisfaction. This suggests that efforts are required to improve the competencies of health professionals to make them more effective while dealing with clients.

Assessing Performance in Health Care Using International Surveys: Are Patient and Clinician Perspectives Complementary or Substitutive

Background:

Over the last decade, international surveys of patients and clinicians have been used to compare health care across countries. Findings from these surveys have been extensively used to create aggregate scores and rankings.

Objective:

To assess the concordance of survey responses provided by patients and clinicians.

Methods:

Analysis of 16 pairs of questions that focused on coordination, organizational factors, and patient-centered competencies from the Commonwealth Fund International Health Policy Survey of older adults (2014) and of primary care physicians (2015). Concordance was assessed by comparing absolute rates and relative rankings.

Results:

In absolute terms, patients and clinicians gave differing responses for questions about coordination of care (patients were more positive) and provision of after-hours care (patients were less positive). In relative terms, country rankings were positively correlated for 5 of 16 question pairs (Spearman ρ > .6 and P < .05).

Conclusion:

Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients’ and clinicians’ perspectives before combining them to create aggregate assessments of performance.