Evaluation of quality indicators for management of the National School Feeding Program in Brazil: a systematic review

Clinical pharmacy key performance indicators for hospital inpatient setting: a systematic review

BACKGROUND

Key performance indicators (KPIs) are quantifiable measures used to monitor the quality of health services. Implementation guidelines for clinical pharmacy services (CPS) do not specify KPIs.

AIM

To assess the quality of the studies that have developed KPIs for CPS in inpatient hospital settings.

METHOD

A systematic review was conducted by searching in Web of Science, Scopus, and PubMed, supplemented with citation analyses and grey literature searches, to retrieve studies addressing the development of KPIs in CPS for hospital inpatients. Exclusions comprised drug- or disease-specific studies and those not written in English, French, Portuguese, or Spanish. The Appraisal of Indicators through Research and Evaluation (AIRE) instrument assessed methodological quality. Domain scores and an overall score were calculated using an equal-weight principle. KPIs were classified into structure, process, and outcome categories. The protocol is available at https://doi.org/10.17605/OSF.IO/KS2G3 .

RESULTS

We included thirteen studies that collectively developed 225 KPIs. Merely five studies scored over 50% on the AIRE instrument, with domains #3 (scientific evidence) and #4 (formulation and usage) displaying low scores. Among the KPIs, 8.4% were classified as structure, 85.8% as process, and 5.8% as outcome indicators. The overall methodological quality did not exhibit a clear association with a major focus on outcomes. None of the studies provided benchmarking reference values.

CONCLUSION

The KPIs formulated for evaluating CPS in hospital settings primarily comprised process measures, predominantly suggested by pharmacists, with inadequate evidence support, lacked piloting or validation, and consequently, were devoid of benchmarking reference values.

Key performance indicators for pharmaceutical services: A systematic review

Background

Key performance indicators (KPIs) are a set of indicators that improve the quality of services provided by pharmacists. They enable the monitoring and evaluation of result progress and optimize decision-making for stakeholders. Currently, there is no systematic review regarding KPIs for pharmaceutical services.

Objectives

To identify and assess the quality of KPIs developed for pharmaceutical services.

Methods

A systematic review was conducted in PubMed, Scopus, EMBASE, and LILACS from the inception of the database until February 5th, 2024. Studies that developed a set of KPIs for pharmaceutical services were included. The indicators were evaluated using the Appraisal of Indicators through Research and Evaluation (AIRE) instrument. Two independent reviewers performed the study selection, data extraction, and quality assessment.

Results

Fifteen studies were included. The studies were conducted in different regions, most of which were developed for clinical services in hospitals or ambulatory settings, and used similar domains for the development of KPIs such as medication review, patient safety, and patient counseling. Literature review combined with the Delphi technique was the method most used by the studies, with content validity by inter-rater agreement. Regarding methodological quality, most studies described information on the purpose, definition, and stakeholders' involvement in the set of KPIs. However, little information was observed on the strategy for risk adjustment, instructions for presenting and interpreting the indicator results, the detailed description of the numerator and denominator, evidence scientific, and the feasibility of the set of KPIs. Only one study achieved a high methodological quality in all domains of the AIRE tool.

Conclusion

Our findings showed the potential of KPIs to monitor and assess pharmacy practice quality. Future studies should expand KPIs for other settings, explore validity evidence of the existing KPIs, provide detailed descriptions of evidence, formulation, and usage, and test their feasibility in daily practice.

Big Data-Planetary Health approach for evaluating the Brazilian Dengue Control Program

OBJECTIVE

This study aims to integrate the concepts of planetary health and big data into the Donabedian model to evaluate the Brazilian dengue control program in the state of São Paulo.

METHODS

Data science methods were used to integrate and analyze dengue-related data, adding context to the structure and outcome components of the Donabedian model. This data, considering the period from 2010 to 2019, was collected from sources such as Department of Informatics of the Unified Health System (DATASUS), the Brazilian Institute of Geography and Statistics (IBGE), WorldClim, and MapBiomas. These data were integrated into a Data Warehouse. K-means algorithm was used to identify groups with similar contexts. Then, statistical analyses and spatial visualizations of the groups were performed, considering socioeconomic and demographic variables, soil, health structure, and dengue cases.

OUTCOMES

Using climate variables, the K-means algorithm identified four groups of municipalities with similar characteristics. The comparison of their indicators revealed certain patterns in the municipalities with the worst performance in terms of dengue case outcomes. Although presenting better economic conditions, these municipalities held a lower average number of community healthcare agents and basic health units per inhabitant. Thus, economic conditions did not reflect better health structure among the three studied indicators. Another characteristic of these municipalities is urbanization. The worst performing municipalities presented a higher rate of urban population and human activity related to urbanization.

CONCLUSIONS

This methodology identified important deficiencies in the implementation of the dengue control program in the state of São Paulo. The integration of several databases and the use of Data Science methods allowed the evaluation of the program on a large scale, considering the context in which activities are conducted. These data can be used by the public administration to plan actions and invest according to the deficiencies of each location.

Desenvolvimento e validação de indicadores de desempenho da gestão do Programa Nacional de Alimentação Escolar

Resumo A avaliação do Programa Nacional de Alimentação Escolar (PNAE) é ação estratégica para garantir a qualidade da gestão. O objetivo deste estudo é desenvolver e validar um instrumento de indicadores de desempenho para avaliação da gestão do PNAE no contexto nacional. Trata-se de um estudo psicométrico de construção e validação de indicadores de desempenho. Foram estabelecidos 12 indicadores, posteriormente, avaliados em painel de especialistas em relação a sete atributos. Para obter consenso foi utilizada a técnica Delphi e a validade de conteúdo foi analisada com a Razão de Validade de Conteúdo. Para a validade de constructo e confiabilidade, nutricionistas que trabalham na execução do PNAE avaliaram a relevância dos indicadores respondendo um questionário online. A taxa de resposta dos especialistas na primeira rodada da técnica Delphi foi de 43,3% (13/30) e de 69,2% (9/13) na segunda rodada. Um novo indicador foi desenvolvido após a avaliação do painel de especialistas. Um total de 281 nutricionistas participaram do estudo da relevância dos indicadores. No geral, a validade de conteúdo e constructo foi alcançada para 13 indicadores. Todos os 13 indicadores se apresentaram relevantes, com potencial para promover a avaliação da gestão operacional do PNAE.