Co-Production Performance Evaluation in Healthcare. A Systematic Review of Methods, Tools and Metrics

A framework for remotely enabled co-design with young people: its development and application with neurodiverse children and their caregivers

Introduction

This paper describes an innovative Framework for Remotely Enabled Co-Design with Young people (FREDY), which details an adaptable four-stage process for generating design concepts with children and other key stakeholders in a naturalistic and inclusive way.

Methods

Recommendations from existing patient engagement and design methodologies were combined to provide research teams with procedures to capture and analyse end-user requirements rapidly. Resulting insights were applied through iterative design cycles to achieve accelerated and user-driven innovation.

Results

Applying this framework with neurodiverse children within the context of healthcare, shows how creative design methods can give rise to new opportunities for co-creating across diverse geographies, abilities, and backgrounds as well as strengthen co-designer approval of the co-design process and resulting product.

Discussion

We summarise key learnings and principles for fostering trust and sustaining participation with remote activities, and facilitating stakeholder design input through continuous collaboration, as well as highlight the potential benefits and challenges of utilising FREDY with neurotypical populations.

Validity, reliability, and readability of single-item and short physical activity questionnaires for use in surveillance: A systematic review

BACKGROUND

Accurate and fast measurement of physical activity is important for surveillance. Even though many physical activity questionnaires (PAQ) are currently used in research, it is unclear which of them is the most reliable, valid, and easy to use. This systematic review aimed to identify existing brief PAQs, describe and compare their measurement properties, and assess their level of readability.

METHODS

We performed a systematic review based on the PRISMA statement. Literature searches were conducted in six scientific databases. Articles were included if they evaluated validity and/or reliability of brief (i.e., with a maximum of three questions) physical activity or exercise questionnaires intended for healthy adults. Due to the heterogeneity of studies, data were summarized narratively. The level of readability was calculated according to the Flesch-Kincaid formula.

RESULTS

In total, 35 articles published in English or Spanish were included, evaluating 32 distinct brief PAQs. The studies indicated moderate to good levels of reliability for the PAQs. However, the majority of results showed weak validity when validated against device-based measurements and demonstrated weak to moderate validity when validated against other PAQs. Most of the assessed PAQs met the criterion of being "short," allowing respondents to complete them in less than one minute either by themselves or with an interviewer. However, only 17 questionnaires had a readability level that indicates that the PAQ is easy to understand for the majority of the population.

CONCLUSIONS

This review identified a variety of brief PAQs, but most of them were evaluated in only a single study. Validity and reliability of short and long questionnaires are found to be at a comparable level, short PAQs can be recommended for use in surveillance systems. However, the methods used to assess measurement properties varied widely across studies, limiting the comparability between different PAQs and making it challenging to identify a single tool as the most suitable. None of the evaluated brief PAQs allowed for the measurement of whether a person fulfills current WHO physical activity guidelines. Future development or adaptation of PAQs should prioritize readability as an important factor to enhance their usability.

Does letting adolescent and young adult inpatients share decisions in choosing the central-line insertion site reduce central-line-associated bloodstream infections? An empty systematic review

To study whether allowing adolescents and young adults (AYA) with chronic or oncologic diseases admitted to tertiary or intensive care units to share decisions in choosing the insertion site for central-venous catheters (CVC) implanted for intravenous therapies or parenteral nutrition reduces central-line-associated and catheter-related bloodstream infections (CLABSI and CRBSI). Following the PRISMA guidelines, we systematically reviewed the literature by searching MEDLINE, Embase, CINAHL, CENTRAL, SCOPUS, Cochrane Library, and Web of Science up to December 2019. According to our aims, the review identified no study that could be included. This empty systematic review on healthcare teams allowing AYA with chronic or oncologic diseases admitted in tertiary or intensive care units to share decisions in choosing the site for implanting CVC prompts further research on clinical pathways on this hot-topic. By considering purportedly risk-taking behaviors in youngsters thus reducing CLABSI and CRBSI, healthcare teams should test specific strategies by engaging AYA empathetically in sharing decisions on the site for implanting CVC to improve quality in health care bundles.

A scoping review of the globally available tools for assessing health research partnership outcomes and impacts

BACKGROUND

Health research partnership approaches have grown in popularity over the past decade, but the systematic evaluation of their outcomes and impacts has not kept equal pace. Identifying partnership assessment tools and key partnership characteristics is needed to advance partnerships, partnership measurement, and the assessment of their outcomes and impacts through systematic study.

OBJECTIVE

To locate and identify globally available tools for assessing the outcomes and impacts of health research partnerships.

METHODS

We searched four electronic databases (Ovid MEDLINE, Embase, CINAHL + , PsychINFO) with an a priori strategy from inception to June 2021, without limits. We screened studies independently and in duplicate, keeping only those involving a health research partnership and the development, use and/or assessment of tools to evaluate partnership outcomes and impacts. Reviewer disagreements were resolved by consensus. Study, tool and partnership characteristics, and emerging research questions, gaps and key recommendations were synthesized using descriptive statistics and thematic analysis.

RESULTS

We screened 36 027 de-duplicated citations, reviewed 2784 papers in full text, and kept 166 studies and three companion reports. Most studies originated in North America and were published in English after 2015. Most of the 205 tools we identified were questionnaires and surveys targeting researchers, patients and public/community members. While tools were comprehensive and usable, most were designed for single use and lacked validity or reliability evidence. Challenges associated with the interchange and definition of terms (i.e., outcomes, impacts, tool type) were common and may obscure partnership measurement and comparison. Very few of the tools identified in this study overlapped with tools identified by other, similar reviews. Partnership tool development, refinement and evaluation, including tool measurement and optimization, are key areas for future tools-related research.

CONCLUSION

This large scoping review identified numerous, single-use tools that require further development and testing to improve their psychometric and scientific qualities. The review also confirmed that the health partnership research domain and its measurement tools are still nascent and actively evolving. Dedicated efforts and resources are required to better understand health research partnerships, partnership optimization and partnership measurement and evaluation using valid, reliable and practical tools that meet partners' needs.

Measurement and outcomes of co-production in health and social care: a systematic review of empirical studies

BACKGROUND

Co-production is promoted as an effective way of improving the quality of health and social care but the diversity of measures used in individual studies makes their outcomes difficult to interpret.

OBJECTIVE

The objective is to explore how empirical studies in health and social care have described the outcomes of co-production projects and how those outcomes were measured.

DESIGN AND METHODS

A scoping review forms the basis for this systematic review. Search terms for the concepts (co-produc* OR coproduc* OR co-design* OR codesign*) and contexts (health OR 'public service* OR "public sector") were used in: CINAHL with Full Text (EBSCOHost), Cochrane Central Register of Controlled trials (Wiley), MEDLINE (EBSCOHost), PsycINFO (ProQuest), PubMed (legacy) and Scopus (Elsevier). There was no date limit. Papers describing the process, original data and outcomes of co-production were included. Protocols, reviews and theoretical, conceptual and psychometric papers were excluded. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses guideline was followed. The Mixed Methods Appraisal Tool underpinned the quality of included papers.

RESULTS

43 empirical studies were included. They were conducted in 12 countries, with the UK representing >50% of all papers. No paper was excluded due to the Mixed Methods Quality Appraisal screening and 60% of included papers were mixed methods studies. The extensive use of self-developed study-specific measures hampered comparisons and cumulative knowledge-building. Overall, the studies reported positive outcomes. Co-production was reported to be positively experienced and provided important learning.

CONCLUSIONS

The lack of common approaches to measuring co-production is more problematic than the plurality of measurements itself. Co-production should be measured from three perspectives: outputs of co-production processes, the experiences of participating in co-production processes and outcomes of co-production. Both self-developed study-specific measures and established measures should be used. The maturity of this research field would benefit from the development and use of reporting guidelines.

Service user involvement in mental health service commissioning, development and delivery: A systematic review of service level outcomes

INTRODUCTION

Service user involvement is increasingly considered essential in mental health service development and delivery. However, the impact of this involvement on services is not well documented. We aimed to understand how user involvement shapes service commissioning, development and delivery, and if/how this leads to improved service-level outcomes.

METHODS

A systematic review of electronic databases (MEDLINE, PsycINFO, CINAHL and EMBASE databases) was undertaken in June and November 2022 for studies that incorporated patient involvement in service development, and reported service-level outcomes. Included studies were synthesised into a logic model based on inputs (method of involvement), activities (changes to service) and outputs (indicators of improvement). PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines were followed when conducting this review.

RESULTS

From 10,901 records identified, nine studies were included, of which six were judged to have used co-production or co-design approaches. Included studies described service user involvement ranging from consultation to co-production. We identified a range of outputs associated with service user involvement in service planning and delivery, and reported these in the form of a logic model. These service-level outputs included improved treatment accessibility, increased referrals and greater service user satisfaction. Longer-term outcomes were rarely reported and hence it was difficult to establish whether outputs are sustained.

CONCLUSION

More extensive forms of involvement, namely, co-design and co-production, were associated with more positive and substantial outputs in regard to service effectiveness than more limited involvement methods. However, lived experience contributions highlighted service perception outputs may be valued more highly by service users than professionals and therefore should be considered equally important when evaluating service user involvement. Although evidence of longer term outcomes was scarce, meaningful involvement of service users in service planning and delivery appeared to improve the quality of mental health services.

PATIENT OR PUBLIC CONTRIBUTION

Members of a lived experience advisory panel contributed to the review findings, which were co-authored by a peer researcher. Review findings were also presented to stakeholders including service users and mental health professionals.

Addressing Health Disparities through Community Participation: A Scoping Review of Co-Creation in Public Health

BACKGROUND

There is general agreement regarding the relevance of community involvement in public health policy, practice, and research to reduce health inequities.

OBJECTIVE

This review aims to analyse the experiences of community engagement in public health actions, with particular attention to methodologies used and how community participation is articulated.

METHOD AND ANALYSIS

We searched the Web of Science, EBSCO, and ProQuest for scientific articles published in peer-reviewed journals. We recorded methodological aspects, the approach to equity, actors that participated in the actions, and participation of the community in different phases (agenda setting, design, implementation, and evaluation).

RESULTS

Of 4331 records, we finally included 31 studies published between 1995 and 2021. Twelve studies referred to Community-Based Participatory Research as the framework used. The actions addressed equity, mainly by tackling economic vulnerability (n = 20, 64%) and racial discrimination (n = 18, 58%). Workshops were the most used method. Participation was frequently observed in the design and implementation phases of the action, but it was reduced to community feedback in the evaluation.

CONCLUSIONS

Co-created public health actions offer the opportunity to reduce health inequity and promote social change; yet, further effort is needed to involve communities in the entire cycle of decision making.

Co-creation in healthcare: framing the outcomes and their determinants

Purpose

Understanding the outcomes of co-creation (CC) in healthcare is increasingly gaining multidisciplinary scientific interest. Although more and more service management scholars have pointed out the benefits of cross-fertilization between the various research fields, the literature on this topic is still scattered and poorly integrated. This study aims to summarize and integrate multiple strands of extant knowledge CC by identifying the outcomes of health CC and the determinants of these outcomes and their relationships.

Design/methodology/approach

A structured literature review was conducted per PRISMA guidelines. A total of 4,189 records were retrieved from the six databases; 1,983 articles were screened, with 161 included in the qualitative thematic analysis.

Findings

This study advances a comprehensive framework for healthcare CC based on a thorough analysis of the outcomes and their determinants, that is, antecedents, management activities and institutional context. Extant research rarely evaluates outcomes from a multidimensional and systemic perspective. Less attention has been paid to the relationship among the CC process elements.

Research limitations/implications

This study offers an agenda to guide future studies on healthcare CC. Highlighting some areas of integration among different disciplines further advances service literature.

Practical implications

The framework offers an operational guide to better shape managerial endeavors to facilitate CC, provide direction and assess multiple outcomes.

Originality/value

This is the first extensive attempt to synthesize and integrate multidisciplinary knowledge on CC outcomes in healthcare settings by adopting a systematic perspective on the overall process.

Evaluating the role and effectiveness of co-produced community-based mental health interventions that aim to reduce suicide among adults: A systematic review

BACKGROUND

Suicide is a major public health risk requiring targeted suicide prevention interventions. The principles of co-production are compatible with tailoring suicide prevention interventions to meet an individual's needs.

AIMS

This review aimed to evaluate the role and effectiveness of co-produced community-based suicide prevention interventions among adults.

METHODS

Four electronic databases (PsycInfo, CINAHL, MEDLINE and web of science) were systematically searched. A narrative synthesis was conducted.

RESULTS

From 590 papers identified through searches, 14 fulfilled the inclusion criteria. Most included studies elicited the views and perspectives of stakeholders in a process of co-design/co-creation of community-based suicide prevention interventions.

CONCLUSION

Stakeholder involvement in the creation of community-based suicide prevention interventions may improve engagement and give voice to those experiencing suicidal crisis. However, there is limited evaluation extending beyond the design of these interventions. Further research is needed to evaluate the long-term outcomes of co-produced community-based suicide prevention interventions.

PATIENT AND PUBLIC INVOLVEMENT

This paper is a systematic review and did not directly involve patients and/or the public. However, the findings incorporate the views and perspectives of stakeholders as reported within the studies included in this review, and the findings may inform the future involvement of stakeholders in the design, development and delivery of community-based suicide prevention interventions for adults.

The Evolution of Balanced Scorecard in Healthcare: A Systematic Review of Its Design, Implementation, Use, and Review

During the last few years, the interest in performance measurement increased within the healthcare sector. Due to the COVID-19 pandemic, healthcare systems needed to boost performance measurement systems to become more resilient and improve their capability in monitoring key performance indicators. Since the 1990s, the Balanced Scorecard (BSC) model has been widely used among private and public organizations as it is the most adopted model to measure performance. The current paper aims at understanding the evolution of BSC in healthcare. The systematic literature review has been carried out by searching keywords according to PRISMA guidelines. By analyzing papers through one classification of BSC adoption phases, the results reveal that studies focused mainly on the BSC design process, rather than BSC implementation, use, or review. However, there is no agreement about the perspectives to be adopted in healthcare. Concerning BSC implementation and use, on one side especially leadership, culture and communication enable the BSC implementation. On the other side, monitoring and strategic decision-making are the most widespread objectives for using BSC. Concerning BSC review, however, the paper highlights a need for additional research. Finally, the paper provides further research opportunities concerning the phases suitable for implementing a BSC in healthcare.

User-centered practices in the eyes of informal caregivers of in-patients with severe acquired brain injury: needs, caring experience, and satisfaction

OBJECTIVE

This study examines the perceived needs, experience, and satisfaction of informal caregivers (ICGs) in in-hospital settings, related to their involvement in the design and delivery of services together with hospital staff, namely co-production.

DESIGN

To obtain a picture of current ICG-staff relationship, a multicenter observational study was carried out. Participants were 75 ICGs recruited in five dedicated in-patient neurorehabilitation wards. Participants answered a self-report questionnaire tapping perceived information/communication needs, emotional/social needs, and their satisfaction; family-centered practices implemented by the staff (namely involving practices and cooperative communication); and ICGs' satisfaction with the service.

RESULTS

Need satisfaction related positively to staff practices aimed at involving IGCs in treatment and training, but not in decision-making. Involving practices concerning treatment also related positively to ICGs' information/communication needs. In addition, the more the staff involved ICGs in decision-making and promoted cooperative communication regarding treatment, the more ICGs felt that their collaboration in the healthcare process was valuable. Finally, all involvement practices and cooperative communication were positively related to ICGs' overall satisfaction with the service.

CONCLUSION

The results of the study help to identify gaps in meeting ICGs' needs and to promote strategies to implement family participation toward co-production in in-hospital settings.