Health systems performance assessment in low-income countries: learning from international experiences

Effect of the data-informed platform for health intervention on the culture of data use for decision-making among district health office staff in North Shewa Zone, Ethiopia: a cluster-randomised controlled trial

BACKGROUND

Similar to other low and middle-income countries, Ethiopia faces limitations in using local health data for decision-making.We aimed to assess the effect of an intervention, namely the data-informed platform for health, on the culture of data-based decision making as perceived by district health office staff in Ethiopia's North Shewa Zone.

METHODS

By designating district health offices as 'clusters', a cluster-randomised controlled trial was implemented. Out of a total of 24 districts in the zone, 12 districts were allocated to intervention arm and the other 12 in the control group arms. In the intervention arm district health office teams were supported in four-monthly cycles of data-driven decision-making over 20 months. This support included: (a) defining problems using a health system framework; (b) reviewing data; (c) considering possible solutions; (d) value-based prioritizing; and (e) a consultative process to develop, commit to, and follow up on action plans. To measure the culture of data use for decision-making in both intervention and control arms, we interviewed 120 health management staff (5 per district office). Using a Likert scale based standard Performance of Routine Information System Management tool, the information is categorized into six domains:- evidence-based decision making, emphasis on data quality, use of information, problem solving, responsibility and motivation. After converting the Likert scale responses into percentiles, difference-in-difference methods were applied to estimate the net effect of the intervention. In intervention districts, analysis of variance was used to summarize variation by staff designation.

RESULTS

The overall decision-making culture in health management staff showed a net improvement of 13% points (95% C.I:9, 18) in intervention districts. The net effect of each of the six domains in turn was an 11% point increase (95% C.I:7, 15) on culture of evidence based decision making, a 16% point increase (95% C.I:8, 24) on emphasis on data quality, a 20% point increase (95% C.I:12, 28) on use of information, a 21% point increase (95% C.I:13, 29) on problem solving, and a 10% point increase (95% C.I:4, 16) on responsibility and motivation. In terms of variation by staff designation within intervention districts, statistically significant differences were observed only for problem solving and responsibility.

CONCLUSION

The data-informed platform for health strategy resulted in a measurable improvement in data use and structured decision-making culture by using existing systems, namely the Performance Monitoring Team meetings. The intervention supported district health offices in identifying and solving problems through a structured process. After further research, DIPH intervention could also be applied to other health administration and facility levels.

TRIAL REGISTRATION

ClinicalTrials.gov ID: NCT05310682, Dated 25/03/ 2022.

The Lifespan Disparity Dataset: An open repository on inequality and polarization in length of life (1950-2021)

Monitoring health is key for identifying priorities in public health planning and improving healthcare services. Life expectancy has conventionally been regarded as a valuable indicator to compare the health status of different populations. However, this measure is simply the mean of the distribution of the length of life and, as such, neglects individual disparities in health outcomes. In this paper, we use life tables from the UN World Population Prospects to develop the most comprehensive dataset of lifespan inequality and polarization for 258 countries and areas for the period 1950-2021. These extensive series on lifespan distributions provide access to crucial information for researchers, practitioners, and the general public, thus contributing to a better understanding of health differences within and between nations.

Making districts functional for universal health coverage attainment: lessons from Ghana

Ensuring the sub national level in the health system can function effectively is central to attainment of health results in countries. However, the current health agenda has not prioritized how districts can deploy their existing resources effectively, to maximize the efficiency equity and effectiveness in their use. Ghana initiated a self-assessment process to understand the functionality of districts to deliver on health results. The assessment was conducted by health managers in 33 districts during August-October 2022 using tools pre-developed by the World Health Organization. Functionality was explored around service provision, oversight, and management capacities, each with defined dimensions and attributes. The objective of the study was to highlight specific functionality improvements needed by districts in terms of investments and access to service delivery in achieving Universal Health Care. The results showed a lack of correlation between functionality and performance as is currently defined in Ghana; a higher functionality of oversight capacity compared to service provision or management capacities; and specifically low functionality for dimensions relating to capacity to make available quality services, responsiveness to beneficiaries and the systems and three structures for health management. The findings highlight the need to shift from quantitative outcome indicator-based performance approaches to measures of total health and wellbeing of beneficiaries. Specific functionality improvements are needed to improve the engagement and answerability to the beneficiaries, investments in access to services, and in building management architecture.

Evaluating Healthcare Performance in Low- and Middle-Income Countries: A Pilot Study on Selected Settings in Ethiopia, Tanzania, and Uganda

The literature reports some experiences regarding the design of integrated healthcare Performance Evaluation Systems (PES) applied in Low- and Middle-income Countries (LMIC). This study describes the design of an integrated and bottom-up PES aimed at evaluating healthcare services delivery in rural settings. The analysis involved four hospitals and their relative health districts in Ethiopia, Tanzania, and Uganda. The evaluation process was undertaken for those indicators that could be evaluated using the same reference standard. The evaluation scores were determined through the international standards identified in the literature or through benchmarking assessment. Both administrative and health data were extracted from the hospitals' registers and District Health Information Systems (DHIS) from 2017 to 2020. We defined 128 indicators: 88 were calculated at the hospital level and 40 at the health district level. The evaluation process was undertaken for 48 indicators. The evaluated indicators are represented using effective graphical tools. In settings characterised by multiple healthcare providers, this framework may contribute to achieving good governance through performance evaluation, benchmarking, and accountability. It may promote evidence-based decision-making in the planning and allocation of resources, thus ultimately fostering quality improvement processes and practices, both at the hospital and health district level.

Geospatial Analysis of Dental Access and Workforce Distribution in Kenya

Background and Objective

One of the major factors affecting access to quality oral healthcare in low- and middle-income countries is the under-supply of the dental workforce. The aim of this study was to use Geographical Information System (GIS) to analyse the distribution and accessibility of the dental workforce and facilities across the Kenyan counties.

Methods

This was a cross-sectional study targeting dental professionals and their practices in Kenya in 2013. Using QGIS 3.16, these data were overlaid with data on population size and urbanization levels. For access measurement, buffers were drawn around each clinic at distances of 2.5, 5, 10 and 20 km, and the population within each determined.

Findings

Nine hundred six dental professionals in 337 dental clinic locations were included in the study. Dentists, community oral health officers (equivalent to dental therapists) and dental technologists comprised 72%, 15% and 12%, respectively. Nairobi county with 100% urbanization and >4000 people/km² had 43% of the workforce and a dentist to population ratio of 1:9,018. Wajir with an urbanization level of 15% and 12 people/km² had no dental facility. Overall, 11%, 19%, 35% and 58% of the Kenyan population were within 2.5, 5, 10 and 20 km radius of a dental clinic respectively.

Conclusion

Maldistribution of dental workforce in Kenya persists, particularly in less urbanized and sparsely populated areas. GIS map production give health planners a better visual picture of areas that are most in need of health care services based on population profiles.

Framework for assessing the performance of the Ghanaian health system: study protocol

Background

Assessing the performance of health systems through quantitative and qualitative methods is recognized as an effective approach to strengthening national health systems. However, while many high-income countries have institutionalized health system performance assessment (HSPA) as an integral component of their respective health systems, few studies on HSPA have been documented in low- and middle-income countries, including Ghana. This study aims at providing a comprehensive framework for periodic assessment of the performance of the entire health system in Ghana.

Methods

The study will have four work packages. First, a structured review will be conducted to identify both international and national HSPA frameworks that could be applied to the Ghanaian context. Second, based on the structured review, an assessment framework tailored to the Ghanaian health system context will be developed. Third, the draft framework will be presented and discussed with experts and stakeholders for its appropriateness, feasibility and acceptability. Finally, the framework will be piloted to assess its effectiveness and suitability for full-scale implementation.

Discussion

Currently, Ghana does not have a full-fledged HSPA tool that provides a holistic health sector-wide approach to assessing health system performance. Thus, developing this HSPA framework for the country will provide a tool for periodic and comprehensive assessment of the performance of the health system, which can be compared with that of other countries. Such a comparison will offer the opportunity for mutual learning and for exploring new options for formulating more effective national health policies. As this is expected to be the first attempt to develop a comprehensive HSPA framework in Ghana, this study will provide a basis for future discussions on how to further develop and implement HSPA programmes in the country.

Towards an Explanation of the Social Value of Health Systems: An Interpretive Synthesis

BACKGROUND

Health systems are complex social systems, and values constitute a central dimension of their complexity. Values are commonly understood as key drivers of health system change, operating across all health systems components and functions. Moreover, health systems are understood to influence and generate social values, presenting an opportunity to harness health systems to build stronger, more cohesive societies. However, there is little investigation (theoretical, conceptual, or empirical) on social values in health policy and systems research (HPSR), particularly regarding the capacity of health systems to influence and generate social values. This study develops an explanatory theory for the 'social value of health systems.'

METHODS

We present the results of an interpretive synthesis of HPSR literature on social values, drawing on a qualitative systematic review, focusing on claims about the relationship between 'health systems' and 'social values.' We combined relational claims extracted from the literature under a common framework in order to generate new explanatory theory.

RESULTS

We identify four mechanisms by which health systems are considered to contribute social value to society: Health systems can: (1) offer a unifying national ideal and build social cohesion, (2) influence and legitimise popular attitudes about rights and entitlements with regard to healthcare and inform citizen's understanding of state responsibilities, (3) strengthen trust in the state and legitimise state authority, and (4) communicate the extent to which the state values various population groups.

CONCLUSION

We conclude that, using a systems-thinking and complex adaptive systems perspective, the above mechanisms can be explained as emergent properties of the dynamic network of values-based connections operating within health systems. We also demonstrate that this theory accounts for how HPSR authors write about the relationship between health systems and social values. Finally, we offer lessons for researchers and policy-makers seeking to bring about values-based change in health systems.

A multidimensional framework for rating health system performance and sustainability: A nine plus one ranking system

Background

Health Care provision in terms of prevention, detection and treatment is primarily dependent on the quality of the hosting Health System. In its health report 2000, the WHO's attempt to assess and rank health systems’ quality Worldwide was heavily criticized. We propose a novel framework for health system performance and ranking using three indicators for three domains; general health system performance, clinical outcome of treatment applied to the main causes of death and health system sustainability domains.

Methods

Each domain was rated as “A – high”, “B – intermediate” or “C – poor” according to the aggregate score values of its three indicators. Hence the highest rank a health system can achieve is “AAA” and the lowest is “CCC”. If there is a need to define a “numerical rank” to further differentiate health systems with similar rating from one another, the total health expenditure per capita per year was used as an additional “number 10” indicator to achieve that level of differentiation. The framework was applied to Health Systems serving most of the World population including China, India, Brazil, USA, Russia, Germany, Japan, UK, France, Singapore and Switzerland. Data pertinent to each indicator was captured from published reports in peer-reviewed journals and/or from official websites. A Delphi survey was conducted for data not available online.

Results

Among the 11 health systems tested, no one scored AAA, Switzerland, France, Germany and Japan scored AAB, Singapore scored ABB, UK scored BBB, USA, Russia and China scored BBC, Brazil scored BCC while India scored CCC. Total health expenditure per capita per year lead to ranking Switzerland first followed by France, Germany, and Japan.

Conclusion

This novel ranking system is a practical and an applicable tool that test health system performance and sustainability. It can be utilized to guide all organizations, people and actions whose primary intent is to promote, restore or maintain health to achieve their targets. An International Health System Ranking database that will be hosted by the Institute of Global, Health, Faculty of Medicine, University of Geneva, Switzerland.

Assessment of public hospitals performance in Gaza governorates using the Pabón Lasso Model

BACKGROUND

Measuring hospital performance can be undertaken in different areas including efficiency, productivity, and quality. Many studies have been conducted using several indicators in this regard.

AIM

This study aimed at measuring efficiency of obstetrics and gynaecology, and paediatrics departments at the public hospitals affiliated with the Palestinian Ministry of Health in 2016, 2017 and 2018.

METHOD

This descriptive study includes 12 hospitals; four (A-D) and eight (E-L) providing obstetrics and gynaecology, and paediatrics services, respectively. Data were collected from health annual reports for the 3-year study period. Pabón Lasso charts were drawn using Microsoft Excel 2013.

RESULTS AND CONCLUSION

During the 3 years, hospitals B, D and C lied in zones 3, 1 and 2, respectively. Hospital A was in Zone 4 in 2016 and 2018, but in Zone 1 in 2017. In 2016, hospitals E, F and H were in Zone 3, while, hospitals I, J, K, and L were in Zone 1. In 2017, hospital G shifted from Zone 2 to Zone 3, and backed to Zone 2 in 2018. Hospital L has moved from Zone 1 to Zone 2 in 2018, whereas other hospitals have remained in the same zones. Inefficiency in hospitals provides an important opportunity for addressing the gaps in a quality and less costly manner. Emphasis antenatal care in primary healthcare is important. Further researches are required to cover other types of hospitals through employing frontier techniques of efficiency measurement.

Towards universal health coverage in the WHO African Region: assessing health system functionality, incorporating lessons from COVID-19

The move towards universal health coverage is premised on having well-functioning health systems, which can assure provision of the essential health and related services people need. Efforts to define ways to assess functionality of health systems have however varied, with many not translating into concrete policy action and influence on system development. We present an approach to provide countries with information on the functionality of their systems in a manner that will facilitate movement towards universal health coverage. We conceptualise functionality of a health system as being a construct of four capacities: access to, quality of, demand for essential services and its resilience to external shocks. We test and confirm the validity of these capacities as appropriate measures of system functionality. We thus provide results for functionality of the 47 countries of the WHO African Region based on this. The functionality of health systems ranges from 34.4 to 75.8 on a 0–100 scale. Access to essential services represents the lowest capacity in most countries of the region, specifically due to poor physical access to services. Funding levels from public and out-of-pocket sources represent the strongest predictors of system functionality, compared with other sources. By focusing on the assessment on the capacities that define system functionality, each country has concrete information on where it needs to focus, in order to improve the functionality of its health system to enable it respond to current needs including achieving universal health coverage, while responding to shocks from challenges such as the 2019 coronavirus disease. This systematic and replicable approach for assessing health system functionality can provide the guidance needed for investing in country health systems to attain universal health coverage goals.

Hospital performance and payment: impact of integrating pay-for-performance on healthcare effectiveness in Lebanon

Background: In 2014 the Lebanese Ministry of Public Health integrated pay-for-performance into setting hospital reimbursement tiers, to provide hospitalization service coverage for the majority of the Lebanese population. This policy was intended to improve effectiveness by decreasing unnecessary hospitalizations, and improve fairness by including risk-adjustment in setting hospital performance scores. Methods: We applied a systematic approach to assess the impact of the new policy on hospital performance. The main impact measure was a national casemix index, calculated across 2011-2016 using medical discharge and surgical procedure codes. A single-group interrupted time series analysis model with Newey ordinary least squares regression was estimated, including adjustment for seasonality, and stratified by case type. Code-level analysis was used to attribute and explain changes in casemix index due to specific diagnoses and procedures. Results: Our final model included 1,353,025 cases across 146 hospitals with a post-intervention lag-time of two months and seasonality adjustment. Among medical cases the intervention resulted in a positive casemix index trend of 0.11% per month (coefficient 0.002, CI 0.001-0.003), and a level increase of 2.25% (coefficient 0.022, CI 0.005-0.039). Trend changes were attributed to decreased cases of diarrhea and gastroenteritis, abdominal and pelvic pain, essential hypertension and fever of unknown origin. A shift from medium to short-stay cases for specific diagnoses was also detected. Level changes were attributed to improved coding practices, particularly for breast cancer, leukemia and chemotherapy. No impact on surgical casemix index was found. Conclusions: The 2014 policy resulted in increased healthcare effectiveness, by increasing the casemix index of hospitals contracted by the Ministry. This increase was mainly attributed to decreased unnecessary hospitalizations and was accompanied by improved medical discharge coding practices. Integration of pay-for-performance within a healthcare system may contribute to improving effectiveness. Effective hospital regulation can be achieved through systematic collection and analysis of routine data.

Frameworks of Performance Measurement in Public Health and Primary Care System: A Scoping Review and Meta-Synthesis

Background:

This study was conducted to synthesize the evidence on the dimensions of performance appraisal of the public health and primary care system through a scoping review and meta-synthesis.

Methods:

The review conducted systematically in 2018 with a scoping review approach. To identify pertinent studies, the following electronic databases were systematically searched until December 20, 2017: Cochrane, ISI Web of Science, PubMed, Scopus, Science Direct, and Embase. Reviewing the studies found on the search bases was carried out in three stages by two persons individually. According to refined studies, the data were extracted to meet the objectives and respond to the research questions. The thematic analysis was used to identify and categorize the dimensions of performance measurement.

Results:

Using this process, 20 studies were eligible for our research. The critical points in measuring the performance of the public health field were classified into eight main domains including leadership and stewardship, funding, resource generation, service delivery, quality, accessibility, efficiency/productivity, and community health status. The differences in measurement frameworks are inevitable. One reason for the differences in the health system performance measurement framework is the differences in the data or data collection, analysis, and reporting. Performance measurement in the field of health, especially primary care, was a multidimensional issue.

Conclusions:

Each of the main dimensions had several sub-criteria, indicating the broadness and complexity of the performance of first-level care providers. Single-dimensional performance measurement could underpin incorrect policies and decisions.

Factors That Influence Data Use to Improve Health Service Delivery in Low- and Middle-Income Countries

BACKGROUND

Health service delivery indicators are designed to reveal how well health services meet a community's needs. Effective use of the data can enable targeted improvements in health service delivery. We conducted a systematic review to identify the factors that influence the use of health service delivery indicators to improve delivery of primary health care services in low- and middle-income settings.

METHODS

We reviewed empirical studies published in 2005 or later that provided evidence on the use of health service delivery data at the primary care level in low- and middle-income countries. We searched Scopus, Medline, the Cochrane Library, and citations of included studies. We also searched the gray literature, using a separate strategy. We extracted information on study design, setting, study population, study objective, key findings, and any identified lessons learned.

RESULTS

Twelve studies met the inclusion criteria. This small number of studies suggests there is insufficient evidence to draw reliable conclusions. However, a content analysis identified the following potentially influential factors, which we classified into 3 categories: governance (leadership, participatory monitoring, regular review of data); production of information (presentation of findings, data quality, qualitative data); and health information system resources (electronic health management information systems, organizational structure, training). Contextual factors and performance-based financing were also each found to have a role; however, discussing these as mediating factors may not be practical in terms of promoting data use.

CONCLUSION

Scant evidence exists regarding factors that influence the use of health service delivery indicators to improve delivery of primary health care services in low- and middle-income countries. However, the existing evidence highlights some factors that may have a role in improving data use. Further research may benefit from comparing data use factors across different types of program indicators or using our classification as a framework for field experiments.

Social values and health systems in health policy and systems research: a mixed-method systematic review and evidence map

Because health systems are conceptualized as social systems, embedded in social contexts and shaped by human agency, values are a key factor in health system change. As such, health systems software-including values, norms, ideas and relationships-is considered a foundational focus of the field of health policy and systems research (HPSR). A substantive evidence-base exploring the influence of software factors on system functioning has developed but remains fragmented, with a lack of conceptual clarity and theoretical coherence. This is especially true for work on 'social values' within health systems-for which there is currently no substantive review available. This study reports on a systematic mixed-methods evidence mapping review on social values within HPSR. The study reaffirms the centrality of social values within HPSR and highlights significant evidence gaps. Research on social values in low- and middle-income country contexts is exceedingly rare (and mostly produced by authors in high-income countries), particularly within the limited body of empirical studies on the subject. In addition, few HPS researchers are drawing on available social science methodologies that would enable more in-depth empirical work on social values. This combination (over-representation of high-income country perspectives and little empirical work) suggests that the field of HPSR is at risk of developing theoretical foundations that are not supported by empirical evidence nor broadly generalizable. Strategies for future work on social values in HPSR are suggested, including: countering pervasive ideas about research hierarchies that prize positivist paradigms and systems hardware-focused studies as more rigorous and relevant to policy-makers; utilizing available social science theories and methodologies; conceptual development to build common framings of key concepts to guide future research, founded on quality empirical research from diverse contexts; and using empirical evidence to inform the development of operationalizable frameworks that will support rigorous future research on social values in health systems.

Monitoring Sustainable Development Goals 3: Assessing the Readiness of Low- and Middle-Income Countries

Background: The Millennium Development Goals (MDGs) availed opportunities for scaling up service coverage but called for stringent monitoring and evaluation (M&E) focusing mainly on MDG related programs. The Sustainable Development Goals 3 (SDGs) and the universal health coverage (UHC) agenda present a broader scope and require more sophisticated M&E systems. We assessed the readiness of low- and middle-income countries to monitor SDG 3.

Methods: Employing mixed methods, we reviewed health sector M&E plans of 6 countries in the World Health Organization (WHO) Africa Region to assess the challenges to M&E, the indicator selection pattern and the extent of multisectoral collaboration. Qualitative data were analysed using content thematic analysis while quantitative data were analysed using Excel.

Results: Challenges to monitoring SDG 3 include weak institutional capacity; fragmentation of M&E functions; inadequate domestic financing; inadequate data availability, dissemination and utilization of M&E products. The total number of indictors in the reviewed plans varied from 38 for Zimbabwe to 235 for Zanzibar. Sixty-nine percent of indicators for the Gambia and 89% for Zanzibar were not classified in any domain in the M&E results chain. Countries lay greater M&E emphasis on service delivery, health systems, maternal and child health as well as communicable diseases with a seeming neglect of the non-communicable diseases (NCDs). Inclusion of SDG 3 indicators only ranged from 48% for Zanzibar to 67% for Kenya. Although monitoring SDG 3 calls for multisectoral collaboration, consideration of the role of other sectors in the M&E plans was either absent or limited to the statistical departments.

Conclusion: There are common challenges confronting M&E at county-level. Countries have omitted key indicators for monitoring components of the SDG 3 targets especially those on NCDs and injuries. The role of other sectors in monitoring SDG 3 targets is not adequately reflected. These could be bottlenecks to tracking progress towards SDG 3 if not addressed. Beyond providing compendium of indicators to guide countries, we advocate for a more binding minimum set of indicators for all countries to which they may add depending on their context. Ministries of Health (MoHs) should prioritise M&E as an important pillar for health service planning and implementation and not as an add-on activity.

Hospital performance and payment: impact of integrating pay-for-performance on healthcare effectiveness in Lebanon

Background: In 2014 the Lebanese Ministry of Public Health integrated pay-for-performance into setting hospital reimbursement tiers, to provide hospitalization service coverage for the majority of the Lebanese population. This policy was intended to improve effectiveness by decreasing unnecessary hospitalizations, and improve fairness by including risk-adjustment in setting hospital performance scores. Methods: We applied a systematic approach to assess the impact of the new policy on hospital performance. The main impact measure was a national casemix index, calculated across 2011-2016 using medical discharge and surgical procedure codes. A single-group interrupted time series analysis model with Newey ordinary least squares regression was estimated, including adjustment for seasonality, and stratified by case type. Code-level analysis was used to attribute and explain changes in casemix index due to specific diagnoses and procedures. Results: Our final model included 1,353,025 cases across 146 hospitals with a post-intervention lag-time of two months and seasonality adjustment. Among medical cases the intervention resulted in a positive casemix index trend of 0.11% per month (coefficient 0.002, CI 0.001-0.003), and a level increase of 2.25% (coefficient 0.022, CI 0.005-0.039). Trend changes were attributed to decreased cases of diarrhea and gastroenteritis, abdominal and pelvic pain, essential hypertension and fever of unknown origin. A shift from medium to short-stay cases for specific diagnoses was also detected. Level changes were attributed to improved coding practices, particularly for breast cancer, leukemia and chemotherapy. No impact on surgical casemix index was found. Conclusions: The 2014 policy resulted in increased healthcare effectiveness, by increasing the casemix index of hospitals contracted by the Ministry. This increase was mainly attributed to decreased unnecessary hospitalizations and was accompanied by improved medical discharge coding practices. Integration of pay-for-performance within a healthcare system may contribute to improving effectiveness. Effective hospital regulation can be achieved through systematic collection and analysis of routine data.

Understanding efficiency and the effect of pay-for-performance across health facilities in Tanzania

BACKGROUND

Ensuring efficient use and allocation of limited resources is crucial to achieving the UHC goal. Performance-based financing that provides financial incentives for health providers reaching predefined targets would be expected to enhance technical efficiency across facilities by promoting an output-oriented payment system. However, there is no study which has systematically assessed efficiency scores across facilities before and after the introduction of pay-for-performance (P4P). This paper seeks to fill this knowledge gap.

METHODS

We used data of P4P evaluation related to healthcare inputs (staff, equipment, medicines) and outputs (outpatient consultations and institutional deliveries) from 75 health facilities implementing P4P in Pwani region, and 75 from comparison districts in Tanzania. We measured technical efficiency using Data Envelopment Analysis and obtained efficiency scores across facilities before and after P4P scheme. We analysed which factors influence technical efficiency by regressing the efficiency scores over a number of contextual factors. We also tested the impact of P4P on efficiency through a difference-in-differences regression analysis.

RESULTS

The overall technical efficiency scores ranged between 0.40 and 0.65 for hospitals and health centres, and around 0.20 for dispensaries. Only 21% of hospitals and health centres were efficient when outpatient consultations and deliveries were considered as output, and <3% out of all facilities were efficient when outpatient consultations only were considered as outputs. Higher efficiency scores were significantly associated with the level of care (hospital and health centre) and wealthier catchment populations. Despite no evidence of P4P effect on efficiency on average, P4P might have improved efficiency marginally among public facilities.

CONCLUSION

Most facilities were not operating at their full capacity indicating potential for improving resource usage. A better understanding of the production process at the facility level and of how different healthcare financing reforms affects efficiency is needed. Effective reforms should improve inputs, outputs but also efficiency.

Frameworks for health systems performance assessment: how comprehensive is Ghana's holistic assessment tool?

Background

Measuring the performance of a health system is an essential requirement in creating systems that generate efficient, equitable, patient-focused, accessible and sustainable results. A fundamental requirement for a performance measurement system is the development of an assessment framework within which specific performance measures could be defined and applied regularly. This paper examines the comprehensiveness of Ghana’s health system assessment framework called the Holistic Assessment Tool in relation to some of the internationally recognized frameworks. The paper also analyzes trends in the performance of the health system to understand whether or not an improvement has been recorded following the adoption and implementation of the Holistic Assessment Tool.

Methods

Mainly secondary data were used in this analysis. Searches were conducted on Google Scholar, PubMed, Scopus and Science Direct between May and July, 2019 for published documents on health system performance assessment. We also obtained unpublished documents from Ghana’s Ministry of Health, Ghana Health Service website, and Ghana Statistical Service database. Descriptive statistics were used to examine trends in the performance of the Ghanaian health system.

Results

While the tool provides a national framework for evaluating the performance of the Ghana Health system in several domains, the Holistic Assessment Tool does not cover key health system domains such as information systems for health, access to essential medicines, and patient-centeredness. Also, the scope of the assessment program seems limited to the evaluation of the Ministry of Health’s annual plans, programs and projects. However, the health system has recorded improvements in population health indicators, such as life expectancy at birth, infant mortality, under-5 mortality, HIV prevalence and disease burden (in terms of disability adjusted life years).

Conclusions

The Holistic Assessment Tool is a useful framework, but needs further refinement, both in scope and in conceptual robustness. Future studies should consider exploring factors influencing performance of the Ghanaian health system. Such information will help in strategizing for better and more improvements.

Effective Project Management of a Pan-African Cancer Research Network: Men of African Descent and Carcinoma of the Prostate (MADCaP)

Purpose

Health research in low- and middle-income countries can generate novel scientific knowledge and improve clinical care, fostering population health improvements to prevent premature death. Project management is a critical part of the success of this research, applying knowledge, skills, tools, and techniques to accomplish required goals. Here, we describe the development and implementation of tools to support a multifaceted study of prostate cancer in Africa, focusing on building strategic and operational capacity.

Methods

Applying a learning organizational framework, we developed and implemented a project management toolkit (PMT) that includes a management process flowchart, a cyclical center-specific schedule of activities, periodic reporting and communication, and center-specific monitoring and evaluation metrics.

Results

The PMT was successfully deployed during year one of the project with effective component implementation occurring through periodic cycles of dissemination and feedback to local center project managers. A specific evaluation was conducted 1 year after study initiation to obtain enrollment data, evaluate individual quality control management plans, and undertake risk log assessments and follow-up. Pilot data obtained identified areas in which centers required mentoring, strengthening, and capacity development. Strategies were implemented to improve project goals and operational capacity through local problem solving, conducting quality control checks and following compliancy with study aims. Moving forward, centers will perform quarterly evaluations and initiate strengthening measures as required.

Conclusion

The PMT has fostered the development of both strategic and operational capacity across project centers. Investment in project management resources is essential to ensuring high-quality, impactful health research in low- and middle-income countries.

Comprehensive measurement of health system performance at district level in India: Generation of a composite index

There have been limited attempts at measurement of health system performance at decentralized levels in low- and middle-income countries. This study was undertaken to develop a composite indicator to measure health system performance at district level in India. Primary data were collected from 377 public health facilities in 21 districts of Haryana state in India using health facility surveys. In addition, 1700 health care providers and 800 clients visiting health facilities were interviewed. Routine health management information system data at district and state level were also analyzed. These data were used for computing 67 input and process indicators covering six health system building blocks. Indicators were normalized and aggregated to generate domain-specific and overall composite health system performance index (HSPI) for each district. Several sensitivity analyses were performed to assess robustness of results. Overall, Panchkula and Ambala districts were found to be the best performing in the state (with HSPI scores of 0.64 and 0.62 out of 1), while Mewat, Faridabad, and Palwal districts had the poorest performance (with HSPI scores of 0.46, 0.49, and 0.48 out of 1). Significant variation in performance was observed for each health system building block. Sensitivity analyses results showed that study findings were robust to variations in methods of aggregation of indicators. Our study provides a framework and methods to measure health system performance at district level in a comprehensive manner. The composite indicator provides a summary snapshot to benchmark performance, while building block and domain scores provide critical information for programmatic action.

Primary health Centres' performance assessment measures in developing countries: review of the empirical literature

Background

It is universally accepted that primary healthcare is essential for achieving public health and that assessment of its performance is critical for continuous improvement. The World Health Organization’s (WHO’s) framework for performance assessment is a comprehensive global standard, but difficult to apply in developing countries because of financial and data constraints. This study aims to review the empirical literature on measures for Primary Health Centre (PHC) performance assessment in developing countries, and compare them for comprehensiveness with the aspects described by the WHO Framework.

Methods

Research articles published in English scientific journals between January 1979 and October 2016 were reviewed systematically. The reporting quality of the article and the quality of the measures were assessed with instruments adapted for the purpose of this study. Data was categorized and described.

Results

Fifteen articles were included in the study out of 4359 articles reviewed. Nine articles used quantitative methods, one article used qualitative methods exclusively and five used mixed methods. Fourteen articles had a good description of the measurement properties. None of the articles presented validity tests of the measures but eleven articles presented measures that were well established. Mostly studies included components of personnel competencies (skilled/ non-skilled) and centre performance (patient satisfaction/cost /efficiency).

Conclusions

In comparison to the WHO framework, the measures in the articles were limited in scope as they did not represent all service components of PHCs. Hence, PHC performance assessment should include system components along with relevant measures of personnel performance beyond knowledge of protocols. Existing measures for PHC performance assessment in developing countries need to be validated and concise measures for neglected aspects need to be developed.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3423-0) contains supplementary material, which is available to authorized users.

A critique of the Uganda district league table using a normative health system performance assessment framework

Background

In 2003 the Uganda Ministry of Health (MoH) introduced the District League Table (DLT) to track district performance. This review of the DLT is intended to add to the evidence base on Health Systems Performance Assessment (HSPA) globally, with emphasis on Low and Middle Income Countries (LMICs), and provide recommendations for adjustments to the current Ugandan reality.

Methods

A normative HSPA framework was used to inform the development of a Key Informant Interview (KII) tool. Thirty Key Informants were interviewed, purposively selected from the Ugandan health system on the basis of having developed or used the DLT. KII data and information from published and grey literature on the Uganda health system was analyzed using deductive analysis.

Results

Stakeholder involvement in the development of the DLT was limited, including MoH officials and development partners, and a few district technical managers. Uganda policy documents articulate a conceptually broad health system whereas the DLT focuses on a healthcare system. The complexity and dynamism of the Uganda health system was insufficiently acknowledged by the HSPA framework. Though DLT objectives and indicators were articulated, there was no conceptual reference model and lack of clarity on the constitutive dimensions. The DLT mechanisms for change were not explicit. The DLT compared markedly different districts and did not identify factors behind observed performance. Uganda lacks a designated institutional unit for the analysis and presentation of HSPA data, and there are challenges in data quality and range.

Conclusions

The critique of the DLT using a normative model supported the development of recommendation for Uganda district HSPA and provides lessons for other LMICs. A similar approach can be used by researchers and policy makers elsewhere for the review and development of other frameworks.

Adjustments in Uganda district HSPA should consider: wider stakeholder involvement with more district managers including political, administrative and technical; better anchoring within the national health system framework; integration of the notion of complexity in the design of the framework; and emphasis on facilitating district decision-making and learning. There is need to improve data quality and range and additional approaches for data analysis and presentation.

A framework to assess management performance in district health systems: a qualitative and quantitative case study in Iran

The aim was to design a district health management performance framework for Iran's healthcare system. The mixed-method study was conducted between September 2015 and May 2016 in Tabriz, Iran. In this study, the indicators of district health management performance were obtained by analyzing the 45 semi-structured surveys of experts in the public health system. Content validity of performance indicators which were generated in qualitative part were reviewed and confirmed based on content validity index (CVI). Also content validity ratio (CVR) was calculated using data acquired from a survey of 21 experts in quantitative part. The result of this study indicated that, initially, 81 indicators were considered in framework of district health management performance and, at the end, 53 indicators were validated and confirmed. These indicators were classified in 11 categories which include: human resources and organizational creativity, management and leadership, rules and ethics, planning and evaluation, district managing, health resources management and economics, community participation, quality improvement, research in health system, health information management, epidemiology and situation analysis. The designed framework model can be used to assess the district health management and facilitates performance improvement at the district level.

Measurement of health system performance at district level: A study protocol

Background

Limited efforts have been observed in low and middle income countries to undertake health system performance assessment at district level. Absence of a comprehensive data collection tool and lack of a standardised single summary measure defining overall performance are some of the main problems. Present study has been undertaken to develop a summary composite health system performance index at district level.

Methods

A broad range of indicators covering all six domains as per building block framework were finalized by an expert panel. The domains were classified into twenty sub-domains, with 70 input and process indicators to measure performance. Seven sub-domains for assessing health system outputs and outcomes were identified, with a total of 28 indicators. Districts in Haryana state from north India were selected for the study. Primary and secondary data will be collected from 378 health facilities, district and state health directorate headquarters. Indicators will be normalized, aggregated to generate composite performance index at district level. Domain specific scores will present the quality of individual building block domains in the public health system. Robustness of the results will be checked using sensitivity analysis.

Expected impact for public health:

The study presents a methodology for comprehensive assessment of all health system domains on basis of input, process, output and outcome indicators which has never been reported from India. Generation of this index will help identify policy and implementation areas of concern and point towards potential solutions. Results may also help understand relationships between individual building blocks and their sub-components.

District decision-making for health in low-income settings: a feasibility study of a data-informed platform for health in India, Nigeria and Ethiopia

Low-resource settings often have limited use of local data for health system planning and decision-making. To promote local data use for decision-making and priority setting, we propose an adapted framework: a data-informed platform for health (DIPH) aimed at guiding coordination, bringing together key data from the public and private sectors on inputs and processes. In working to transform this framework from a concept to a health systems initiative, we undertook a series of implementation research activities including background assessment, testing and scaling up of the intervention. This first paper of four reports the feasibility of the approach in a district health systems context in five districts of India, Nigeria and Ethiopia. We selected five districts using predefined criteria and in collaboration with governments. After scoping visits, an in-depth field visit included interviews with key health stakeholders, focus group discussions with service-delivery staff and record review. For analysis, we used five dimensions of feasibility research based on the TELOS framework: technology and systems, economic, legal and political, operational and scheduling feasibility. We found no standardized process for data-based district level decision-making, and substantial obstacles in all three countries. Compared with study areas in Ethiopia and Nigeria, the health system in Uttar Pradesh is relatively amenable to the DIPH, having relative strengths in infrastructure, technological and technical expertise, and financial resources, as well as a district-level stakeholder forum. However, a key challenge is the absence of an effective legal framework for engagement with India’s extensive private health sector. While priority-setting may depend on factors beyond better use of local data, we conclude that a formative phase of intervention development and pilot-testing is warranted as a next step.

Harmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue?

BACKGROUND

Health sector and programme performance assessments provide a rich source of contextual data directly linked to implementation of programmes and can inform health policy dialogue, planning and resource allocation. In seeking to maximise this opportunity, there are challenges to overcome. A meeting convened by the World Health Organization African Region discussed the strengths, weaknesses and challenges to harmonising and standardising health sector and programme performance assessments, as well as use of evidence from such processes in decision making. This article synthesises the deliberations which emerged from the meeting. Discussing these in light of other literature we propose practical options to standardising health sector and programme performance assessment and improve realisation of using evidence in decision making.

DISCUSSION

Use of evidence generated from health sector and programme performance assessments into regular country processes of sectoral monitoring, dialogue and policy modification is crucial. However, this process faces several challenges. Identified challenges were categorised under several themes, namely the weak institutional capacities for monitoring and evaluation in reference to weak health information systems, a lack of tools and skills, and weak accountability mechanisms; desynchronised planning timeframes between programme and overall health sector strategies; inadequate time to undertake comprehensive and good quality performance assessment; weak mechanisms for following up on implementation of recommendations; lack of effective stakeholder participation; and divergent political aspirations.

CONCLUSION

The question of what performance assessment is for in a country must be asked and answered clearly if the utility of these processes is to be realised. Standardising programme and sector reviews offers numerable opportunities that need to be maximised. Identified challenges need to be overcome through strengthened Ministry of Health leadership, effective stakeholder engagement and institutionalising follow-up mechanisms for agreed recommendations. In addition, health sector performance assessments need to be institutionalised as part of the accountability mechanism, and they must be planned for and funding secured within annual budget and medium term expenditure frameworks.

Harmonisation and standardisation of health sector and programme reviews and evaluations - how can they better inform health policy dialogue?

BACKGROUND

Health sector and programme performance assessments provide a rich source of contextual data directly linked to implementation of programmes and can inform health policy dialogue, planning and resource allocation. In seeking to maximise this opportunity, there are challenges to overcome. A meeting convened by the World Health Organization African Region discussed the strengths, weaknesses and challenges to harmonising and standardising health sector and programme performance assessments, as well as use of evidence from such processes in decision making. This article synthesises the deliberations which emerged from the meeting. Discussing these in light of other literature we propose practical options to standardising health sector and programme performance assessment and improve realisation of using evidence in decision making.

DISCUSSION

Use of evidence generated from health sector and programme performance assessments into regular country processes of sectoral monitoring, dialogue and policy modification is crucial. However, this process faces several challenges. Identified challenges were categorised under several themes, namely the weak institutional capacities for monitoring and evaluation in reference to weak health information systems, a lack of tools and skills, and weak accountability mechanisms; desynchronised planning timeframes between programme and overall health sector strategies; inadequate time to undertake comprehensive and good quality performance assessment; weak mechanisms for following up on implementation of recommendations; lack of effective stakeholder participation; and divergent political aspirations.

CONCLUSION

The question of what performance assessment is for in a country must be asked and answered clearly if the utility of these processes is to be realised. Standardising programme and sector reviews offers numerable opportunities that need to be maximised. Identified challenges need to be overcome through strengthened Ministry of Health leadership, effective stakeholder engagement and institutionalising follow-up mechanisms for agreed recommendations. In addition, health sector performance assessments need to be institutionalised as part of the accountability mechanism, and they must be planned for and funding secured within annual budget and medium term expenditure frameworks.

Decentralization and health system performance - a focused review of dimensions, difficulties, and derivatives in India

Introduction

One of the principal goals of any health care system is to improve health through the provision of clinical and public health services. Decentralization as a reform measure aims to improve inputs, management processes and health outcomes, and has political, administrative and financial connotations. It is argued that the robustness of a health system in achieving desirable outcomes is contingent upon the width and depth of ‘decision space’ at the local level. Studies have used different approaches to examine one or more facets of decentralization and its effect on health system functioning; however, lack of consensus on an acceptable framework is a critical gap in determining its quantum and quality. Theorists have resorted to concepts of ‘trust’, ‘convenience’ and ‘mutual benefits’ to explain, define and measure components of governance in health. In the emerging ‘continuum of health services’ model, the challenge lies in identifying variables of performance (fiscal allocation, autonomy at local level, perception of key stakeholders, service delivery outputs, etc.) through the prism of decentralization in the first place, and in establishing directed relationships among them.

Methods

This focused review paper conducted extensive web-based literature search, using PubMed and Google Scholar search engines. After screening of key words and study objectives, we retrieved 180 articles for next round of screening. One hundred and four full articles (three working papers and 101 published papers) were reviewed in totality. We attempted to summarize existing literature on decentralization and health systems performance, explain key concepts and essential variables, and develop a framework for further scientific scrutiny. Themes are presented in three separate segments of dimensions, difficulties and derivatives.

Results

Evaluation of local decision making and its effect on health system performance has been studied in a compartmentalized manner. There is sparse evidence about innovations attributable to decentralization. We observed that in India, there is very scant evaluative study on the subject. We didn’t come across a single study examining the perception and experiences of local decision makers about the opportunities and challenges they faced. The existing body of evidences may be inadequate to feed into sound policy making. The principles of management hinge on measurement of inputs, processes and outputs. In the conceptual framework we propose three levels of functions (health systems functions, management functions and measurement functions) being intricately related to inputs, processes and outputs. Each level of function encompasses essential elements derived from the synthesis of information gathered through literature review and non-participant observation. We observed that it is difficult to quantify characteristics of governance at institutional, system and individual levels except through proxy means.

Conclusion

There is an urgent need to sensitize governments and academia about how best more objective evaluation of ‘shared governance’ can be undertaken to benefit policy making. The future direction of enquiry should focus on context-specific evidence of its effect on the entire spectrum of health system, with special emphasis on efficiency, community participation, human resource management and quality of services.

Globalization and Health: developing the journal to advance the field

Founded in 2005, Globalization and Health was the first open access global health journal. The journal has since expanded the field, and its influence, with the number of downloaded papers rising 17-fold, to over 4 million. Its ground-breaking papers, leading authors -including a Nobel Prize winner- and an impact factor of 2.25 place it among the top global health journals in the world. To mark the ten years since the journal's founding, we, members of the current editorial board, undertook a review of the journal's progress over the last decade. Through the application of an inductive thematic analysis, we systematically identified themes of research published in the journal from 2005 to 2014. We identify key areas the journal has promoted and consider these in the context of an existing framework, identify current gaps in global health research and highlight areas we, as a journal, would like to see strengthened.

A comparison of hierarchical cluster analysis and league table rankings as methods for analysis and presentation of district health system performance data in Uganda

In 2003, the Uganda Ministry of Health introduced the district league table for district health system performance assessment. The league table presents district performance against a number of input, process and output indicators and a composite index to rank districts. This study explores the use of hierarchical cluster analysis for analysing and presenting district health systems performance data and compares this approach with the use of the league table in Uganda. Ministry of Health and district plans and reports, and published documents were used to provide information on the development and utilization of the Uganda district league table. Quantitative data were accessed from the Ministry of Health databases. Statistical analysis using SPSS version 20 and hierarchical cluster analysis, utilizing Wards' method was used. The hierarchical cluster analysis was conducted on the basis of seven clusters determined for each year from 2003 to 2010, ranging from a cluster of good through moderate-to-poor performers. The characteristics and membership of clusters varied from year to year and were determined by the identity and magnitude of performance of the individual variables. Criticisms of the league table include: perceived unfairness, as it did not take into consideration district peculiarities; and being oversummarized and not adequately informative. Clustering organizes the many data points into clusters of similar entities according to an agreed set of indicators and can provide the beginning point for identifying factors behind the observed performance of districts. Although league table ranking emphasize summation and external control, clustering has the potential to encourage a formative, learning approach. More research is required to shed more light on factors behind observed performance of the different clusters. Other countries especially low-income countries that share many similarities with Uganda can learn from these experiences.

Subnational benchmarking of health systems performance in Africa using health outcome and coverage indicators

National health systems performance (HSP) assessments and benchmarking are critical to understanding how well the delivery of healthcare meets the needs of citizens. Benchmarking HSP has often been done between countries to inform the global public health space. However, its impact is likely to be far greater when implemented sub-nationally to inform actual decisions on resource allocations and performance improvements, especially in high disease burden, low-income countries, where the resource envelope available for health is inadequate. In their study, Roberts and colleagues assemble, analyse and map a minimum set of health intervention and outcome indicators from 1990-2011 to assess and benchmark HSP across the 11 regions of Uganda. This is the first empirical sub-national HSP benchmarking study in the country and the results have potentially important health system policy implications. Please see related research: http://www.biomedcentral.com/1741-7015/13/285.

Developing a Performance Assessment Framework and Indicators for Communicable Disease Management in Natural Disasters

INTRODUCTION

Communicable disease management (CDM) is an important component of disaster public health response operations. However, there is a lack of any performance assessment (PA) framework and related indicators for the PA. This study aimed to develop a PA framework and indicators in CDM in disasters.

METHODS

In this study, a series of methods were used. First, a systematic literature review (SLR) was performed in order to extract the existing PA frameworks and indicators. Then, using a qualitative approach, some interviews with purposively selected experts were conducted and used in developing the PA framework and indicators. Finally, the analytical hierarchy process (AHP) was used for weighting of the developed indicators.

RESULTS

The input, process, products, and outcomes (IPPO) framework was found to be an appropriate framework for CDM PA. Seven main functions were revealed to CDM during disasters. Forty PA indicators were developed for the four categories.

CONCLUSION

There is a lack of any existing PA framework in CDM in disasters. Thus, in this study, a PA framework (IPPO framework) was developed for the PA of CDM in disasters through a series of methods. It can be an appropriate framework and its indicators could measure the performance of CDM in disasters.

[Integrated Quality Management System (IQMS): a model for improving the quality of reproductive health care in rural Kenya]

OBJECTIVES

To develop a model aiming to improve the quality of services for reproductive health care in rural Kenya and designed to measure the quality of reproductive health services in such a way that allows these services to identify measures for improving their performance.

METHODS

The Integrated Quality Management System (IQMS) was developed on the basis of a pre-existing and validated model for quality promotion, namely the European Practice Assessment (EPA). The methodology for quality assessment and feedback of assessment results to the service teams was adopted from the EPA model. Quality assessment methodology included data assessment through staff, patient surveys and service visitation. Quality is assessed by indicators, and so indicators had to be developed that were appropriate for assessing reproductive health care in rural Kenya. A search of the Kenyan and international literature was conducted to identify potential indicators. These were then rated for their relevance and clarity by a panel of Kenyan experts.

RESULTS

260 indicators were rated as relevant and assigned to 29 quality dimensions and 5 domains. The implementation of IQMS in ten facilities showed that IQMS is a feasible model for assessing the quality of reproductive health services in rural Kenya. IQMS enables these services to identify quality improvement targets and necessary improvement measures. Both strengths and limitations of IQMS will be discussed.

Evaluating the use of locally-based health facility assessments in Afghanistan: a pilot study of a novel research method

BACKGROUND

Through the Balanced Scorecard program there have been independent, annual and nationwide assessments of the Afghan health system from 2004 to 2013. During this period, Afghanistan remained in a dynamic state of conflict, requiring innovative approaches to health service evaluation in insecure areas. The primary objective of this pilot study was to evaluate the reliability of health facility assessments conducted by a novel, locally-based data collection method compared to a standard survey team.

METHODS

In this cross-sectional study, one standard survey team of clinicians and multiple rapidly trained locally-based survey teams of teachers conducted health facility assessments in Badghis province, Afghanistan from March - August, 2010. Outpatient facilities covered under the country's Basic Package of Health Services were eligible for inclusion. Both approaches attempted to survey as many health facilities as safely possible, up to 25 total facilities per method. Each facility assessed was scored on 23 health services indicators used to evaluate performance in the annual Balanced Scorecard national assessment. For facilities assessed by both survey methods, the indicator scores produced by each method were compared using Spearman's correlation coefficients and linear regression analysis with generalized estimating equations.

RESULTS

The standard survey team was able to assess 11 facilities; the locally-based approach was able to assess these 11 facilities, as well as 13 additional facilities in areas of greater insecurity. Among the 11 facilities assessed by both approaches, 19 of 23 indicators were statistically similar by survey method (p < .05). Spearman's coefficients varied widely from (-0.39) to (0.71). The differences were greatest for items requiring specialized data collector knowledge on reviewing patient records, patient examination and counseling, and health worker reported satisfaction.

CONCLUSIONS

This pilot study of a novel method of data collection in health facility assessments showed that an approach using locally-based survey teams provided markedly increased access to areas of insecurity. Though analysis was limited by small sample size, indicator scores used for facility evaluation were relatively comparable overall, but less reliable for items requiring clinical knowledge or when asking health worker opinions, suggesting that alternative approaches may be needed to assess these parameters in insecure environments.