1
|
Gustafsson J, Ray C, Lehto E, Roos E, Lehto R. Role of early childhood educators' demographic characteristics and perceived work environment in implementation of a preschool health promotion intervention. Arch Public Health 2023; 81:127. [PMID: 37420242 DOI: 10.1186/s13690-023-01133-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 06/14/2023] [Indexed: 07/09/2023] Open
Abstract
BACKGROUND Research has indicated that the effectiveness of intervention programs is affected by how well these programs are implemented, but key gaps remain in our understanding of the factors that promote or inhibit implementation. This study examined how demographic characteristics and perceived work environment among early childhood educators were associated with implementation outcomes of the Increased Health and Wellbeing in Preschools (DAGIS) intervention, which was conducted as a cluster randomized trial. METHODS Participants included 101 educators from 32 intervention preschool classrooms. Data were analyzed at the classroom level, as the DAGIS intervention was delivered in preschool classrooms consisting of several educators instead of individual implementers. Linear regression was used to estimate the associations of educators' demographic characteristics and perceived work environment with different aspects of implementation (i.e., dose delivered; dose received - exposure; dose received - satisfaction; and perceived quality, as well as a total sum score based on these four dimensions). Municipality was controlled in the adjusted models. RESULTS Findings indicated that having a higher proportion of educators with a Bachelor's or Master's degree in education within the classroom was associated with higher dose received - exposure and higher total degree of implementation, and the significance of the models was unaffected by adjustment for municipality. Moreover, having a higher proportion of educators younger than 35 years within the classroom was associated with higher dose received - exposure. However, the association was non-significant when adjusted for municipality. No other educator factor (i.e., work experience in years and perceived support from coworkers, group work, and innovative climate) predicted implementation outcomes. CONCLUSIONS Higher educational attainment and younger age among educators at the classroom level were associated with higher scores for some of the implementation outcomes. Educators' work experience in years at the current preschool and in early childhood education, support from coworkers, group work, and innovative climate were not significantly associated with any implementation outcomes. Future research should explore ways to improve educators' implementation of interventions aimed at promoting children's health behaviors.
Collapse
Affiliation(s)
- Jasmine Gustafsson
- Public Health Research Program, Folkhälsan Research Center, Helsinki, Finland.
- Faculty of Social Sciences, University of Helsinki, Helsinki, Finland.
| | - Carola Ray
- Public Health Research Program, Folkhälsan Research Center, Helsinki, Finland
- Department of Food and Nutrition, University of Helsinki, Helsinki, Finland
| | - Elviira Lehto
- Public Health Research Program, Folkhälsan Research Center, Helsinki, Finland
- Department of Teacher Education, University of Helsinki, Helsinki, Finland
| | - Eva Roos
- Public Health Research Program, Folkhälsan Research Center, Helsinki, Finland
- Department of Food Studies, Nutrition and Dietetics, Uppsala University, Uppsala, Sweden
- Department of Public Health, University of Helsinki, Helsinki, Finland
| | - Reetta Lehto
- Public Health Research Program, Folkhälsan Research Center, Helsinki, Finland
| |
Collapse
|
2
|
Sweetnam C, Goulding L, Davis RE, Khadjesari Z, Boaz A, Healey A, Sevdalis N, Bakolis I, Hull L. Development and psychometric evaluation of the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool: a study protocol. BMJ Open 2022; 12:e061209. [PMID: 36526311 PMCID: PMC9764655 DOI: 10.1136/bmjopen-2022-061209] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Accepted: 09/29/2022] [Indexed: 12/23/2022] Open
Abstract
INTRODUCTION The need for quantitative criteria to appraise the quality of implementation research has recently been highlighted to improve methodological rigour. The Implementation Science Research development (ImpRes) tool and supplementary guide provide methodological guidance and recommendations on how to design high-quality implementation research. This protocol reports on the development of the Implementation Science Research Project Appraisal Criteria (ImpResPAC) tool, a quantitative appraisal tool, developed based on the structure and content of the ImpRes tool and supplementary guide, to evaluate the conceptual and methodological quality of implementation research. METHODS AND ANALYSIS This study employs a three-stage sequential mixed-methods design. During stage 1, the research team will map core domains of the ImpRes tool, guidance and recommendations contained in the supplementary guide and within the literature, to ImpResPAC. In stage 2, an international multidisciplinary expert group, recruited through purposive sampling, will inform the refinement of ImpResPAC, including content, scoring system and user instructions. In stage 3, an extensive psychometric evaluation of ImpResPAC, that was created in stage 1 and refined in stage 2, will be conducted. The scaling assumptions (inter-item and item-total correlations), reliability (internal consistency, inter-rater) and validity (construct and convergent validity) will be investigated by applying ImpResPAC to 50 protocols published in Implementation Science. We envisage developing ImpResPAC in this way will provide implementation research stakeholders, primarily grant reviewers and educators, a comprehensive, transparent and fair appraisal of the conceptual and methodological quality of implementation research, increasing the likelihood of funding research that will generate knowledge and contribute to the advancement of the field. ETHICS AND DISSEMINATION This study will involve human participants. This study has been registered and minimal risk ethical clearance granted by The Research Ethics Office, King's College London (reference number MRA-20/21-20807). Participants will receive written information on the study via email and will provide e-consent if they wish to participate. We will use traditional academic modalities of dissemination (eg, conferences and publications).
Collapse
Affiliation(s)
- Chloe Sweetnam
- Neurology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Lucy Goulding
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| | - Rachel E Davis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| | - Zarnie Khadjesari
- Behavioural and Implementation Science Research Group, School of Health Sciences, University of East Anglia, Norwich, UK
| | - Annette Boaz
- Department of Health Services Research and Policy, London School of Hygiene & Tropical Medicine, London, UK
| | - Andy Healey
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
- King's Health Economics, Institute of Psychiatry, Psychology & Neuroscience, King's College London, London, UK
| | - Nick Sevdalis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| | - Ioannis Bakolis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
- Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Louise Hull
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| |
Collapse
|
3
|
Hall A, Shoesmith A, Doherty E, McEvoy B, Mettert K, Lewis CC, Wolfenden L, Yoong S, Kingsland M, Shelton RC, Wiltsey Stirman S, Imad N, Sutherland R, Nathan N. Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review. Implement Sci 2022; 17:81. [PMID: 36514059 PMCID: PMC9746194 DOI: 10.1186/s13012-022-01252-1] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2022] [Accepted: 10/19/2022] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings. METHODS Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed. RESULTS A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants. CONCLUSIONS This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field. TRIAL REGISTRATION This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.
Collapse
Affiliation(s)
- Alix Hall
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Adam Shoesmith
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Emma Doherty
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Brydie McEvoy
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Kayne Mettert
- grid.488833.c0000 0004 0615 7519Kaiser Permanente Washington Health Research Institute, Seattle, USA
| | - Cara C. Lewis
- grid.488833.c0000 0004 0615 7519Kaiser Permanente Washington Health Research Institute, Seattle, USA ,grid.34477.330000000122986657Department of Psychology, University of Washington, Seattle, USA
| | - Luke Wolfenden
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Serene Yoong
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia ,grid.1021.20000 0001 0526 7079School of Health Sciences and Social Development, Deakin University, Melbourne, Victoria Australia
| | - Melanie Kingsland
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Rachel C. Shelton
- grid.21729.3f0000000419368729Mailman School of Public Health, Department of Sociomedical Sciences, Columbia University, New York, New York USA
| | - Shannon Wiltsey Stirman
- grid.168010.e0000000419368956Dissemination and Training Division, National Center for PTSD and Department of Psychiatry and Behavioural Sciences, Stanford Medicine, Stanford University, Palo Alto, California USA
| | - Noor Imad
- grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia ,grid.1027.40000 0004 0409 2862School of Health Sciences, Department of Nursing and Allied Health, Swinburne University of Technology, Hawthorn, Victoria Australia
| | - Rachel Sutherland
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| | - Nicole Nathan
- grid.266842.c0000 0000 8831 109XSchool of Medicine and Public Health, The University of Newcastle, Locked Bag 10 Wallsend, Callaghan, NSW Australia ,grid.266842.c0000 0000 8831 109XPriority Research Centre for Health Behaviour, The University of Newcastle, Callaghan, NSW Australia ,grid.413648.cHunter Medical Research Institute, New Lambton Heights, New South Wales Australia ,grid.3006.50000 0004 0438 2042Hunter New England Population Health, Hunter New England Local Health District, Wallsend, NSW Australia
| |
Collapse
|
4
|
Pallesen H, Honore H, Aadal L, Kjeldsen SS. Improved active participation among patients with severe acquired brain injury in early rehabilitation: Implementation of a professional competence development intervention. Work 2022; 72:1381-1396. [DOI: 10.3233/wor-205343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
BACKGROUND: In 2019, an educational programme was implemented in a sub-acute in-hospital neurorehabilitation clinic for patients with severe acquired brain injury (sABI). The programme was initiated to enhance staff competencies related to identifying and improving active participation among patients with sABI. OBJECTIVE: The purpose was to evaluate the implementation effectiveness of the educational programme. METHODS: Mixed methods were chosen to assess implementation effectiveness as perceived by staff and patients. RESULTS: A survey of the professional’s experience showed an increase in perceived competence after each completed seminar and from before the first seminar to after the last completed seminar. These results were confirmed and elaborated through staff focus group interviews. The proportion of patients achieving active participation increased from 45% before to 75% after implementation (six of eight patients). CONCLUSION: Exploring the implementation effectiveness of the educational programme seemed clinically valuable and showed a promising and probable effect of an implementation process.
Collapse
Affiliation(s)
- Hanne Pallesen
- Hammel Neurorehabilitation Centre, University of Aarhus, Aarhus, Denmark
- University Research Clinic8 (RM), University of Aarhus, Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| | - Helene Honore
- Hammel Neurorehabilitation Centre, University of Aarhus, Aarhus, Denmark
- University Research Clinic8 (RM), University of Aarhus, Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| | - Lena Aadal
- Hammel Neurorehabilitation Centre, University of Aarhus, Aarhus, Denmark
- University Research Clinic8 (RM), University of Aarhus, Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| | - Simon Svanborg Kjeldsen
- Hammel Neurorehabilitation Centre, University of Aarhus, Aarhus, Denmark
- University Research Clinic8 (RM), University of Aarhus, Aarhus, Denmark
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| |
Collapse
|
5
|
Khadjesari Z, Boufkhed S, Vitoratou S, Schatte L, Ziemann A, Daskalopoulou C, Uglik-Marucha E, Sevdalis N, Hull L. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implement Sci 2020; 15:66. [PMID: 32811517 PMCID: PMC7433178 DOI: 10.1186/s13012-020-01027-6] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Accepted: 07/29/2020] [Indexed: 01/05/2023] Open
Abstract
BACKGROUND Implementation research aims to facilitate the timely and routine implementation and sustainment of evidence-based interventions and services. A glaring gap in this endeavour is the capability of researchers, healthcare practitioners and managers to quantitatively evaluate implementation efforts using psychometrically sound instruments. To encourage and support the use of precise and accurate implementation outcome measures, this systematic review aimed to identify and appraise studies that assess the measurement properties of quantitative implementation outcome instruments used in physical healthcare settings. METHOD The following data sources were searched from inception to March 2019, with no language restrictions: MEDLINE, EMBASE, PsycINFO, HMIC, CINAHL and the Cochrane library. Studies that evaluated the measurement properties of implementation outcome instruments in physical healthcare settings were eligible for inclusion. Proctor et al.'s taxonomy of implementation outcomes was used to guide the inclusion of implementation outcomes: acceptability, appropriateness, feasibility, adoption, penetration, implementation cost and sustainability. Methodological quality of the included studies was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist. Psychometric quality of the included instruments was assessed using the Contemporary Psychometrics checklist (ConPsy). Usability was determined by number of items per instrument. RESULTS Fifty-eight publications reporting on the measurement properties of 55 implementation outcome instruments (65 scales) were identified. The majority of instruments assessed acceptability (n = 33), followed by appropriateness (n = 7), adoption (n = 4), feasibility (n = 4), penetration (n = 4) and sustainability (n = 3) of evidence-based practice. The methodological quality of individual scales was low, with few studies rated as 'excellent' for reliability (6/62) and validity (7/63), and both studies that assessed responsiveness rated as 'poor' (2/2). The psychometric quality of the scales was also low, with 12/65 scales scoring 7 or more out of 22, indicating greater psychometric strength. Six scales (6/65) rated as 'excellent' for usability. CONCLUSION Investigators assessing implementation outcomes quantitatively should select instruments based on their methodological and psychometric quality to promote consistent and comparable implementation evaluations. Rather than developing ad hoc instruments, we encourage further psychometric testing of instruments with promising methodological and psychometric evidence. SYSTEMATIC REVIEW REGISTRATION PROSPERO 2017 CRD42017065348.
Collapse
Affiliation(s)
- Zarnie Khadjesari
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK.
- Behavioural and Implementation Science research group, School of Health Sciences, University of East Anglia, Edith Cavell Building, Norwich Research Park, Norwich, NR4 7TJ, UK.
| | - Sabah Boufkhed
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Silia Vitoratou
- Psychometrics and Measurement Lab, Biostatistics and Health Informatics Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Laura Schatte
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Alexandra Ziemann
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
- Centre for Healthcare Innovation Research, City, University of London, Northampton Square, London, EC1V 0HB, UK
| | - Christina Daskalopoulou
- Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Eleonora Uglik-Marucha
- Psychometrics and Measurement Lab, Biostatistics and Health Informatics Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Nick Sevdalis
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| | - Louise Hull
- Centre for Implementation Science, Health Service and Population Research Department, Institute of Psychiatry, Psychology and Neuroscience, King's College London, 16 De Crespigny Park, London, SE5 8AF, UK
| |
Collapse
|
6
|
Hull L, Goulding L, Khadjesari Z, Davis R, Healey A, Bakolis I, Sevdalis N. Designing high-quality implementation research: development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implement Sci 2019; 14:80. [PMID: 31412887 PMCID: PMC6693182 DOI: 10.1186/s13012-019-0897-z] [Citation(s) in RCA: 57] [Impact Index Per Article: 11.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2018] [Accepted: 04/15/2019] [Indexed: 12/26/2022] Open
Abstract
Background Designing implementation research can be a complex and daunting task, especially for applied health researchers who have not received specialist training in implementation science. We developed the Implementation Science Research Development (ImpRes) tool and supplementary guide to address this challenge and provide researchers with a systematic approach to designing implementation research. Methods A multi-method and multi-stage approach was employed. An international, multidisciplinary expert panel engaged in an iterative brainstorming and consensus-building process to generate core domains of the ImpRes tool, representing core implementation science principles and concepts that researchers should consider when designing implementation research. Simultaneously, an iterative process of reviewing the literature and expert input informed the development and content of the tool. Once consensus had been reached, specialist expert input was sought on involving and engaging patients/service users; and economic evaluation. ImpRes was then applied to 15 implementation and improvement science projects across the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South London, a research organisation in London, UK. Researchers who applied the ImpRes tool completed an 11-item questionnaire evaluating its structure, content and usefulness. Results Consensus was reached on ten implementation science domains to be considered when designing implementation research. These include implementation theories, frameworks and models, determinants of implementation, implementation strategies, implementation outcomes and unintended consequences. Researchers who used the ImpRes tool found it useful for identifying project areas where implementation science is lacking (median 5/5, IQR 4–5) and for improving the quality of implementation research (median 4/5, IQR 4–5) and agreed that it contained the key components that should be considered when designing implementation research (median 4/5, IQR 4–4). Qualitative feedback from researchers who applied the ImpRes tool indicated that a supplementary guide was needed to facilitate use of the tool. Conclusions We have developed a feasible and acceptable tool, and supplementary guide, to facilitate consideration and incorporation of core principles and concepts of implementation science in applied health implementation research. Future research is needed to establish whether application of the tool and guide has an effect on the quality of implementation research.
Collapse
Affiliation(s)
- Louise Hull
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK.
| | - Lucy Goulding
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| | - Zarnie Khadjesari
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK.,School of Health Sciences, University of East Anglia, Norwich Research Park, Norwich, UK
| | - Rachel Davis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| | - Andy Healey
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK.,King's Health Economics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Ioannis Bakolis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK.,Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Nick Sevdalis
- Centre for Implementation Science, Health Service and Population Research Department, King's College London, London, UK
| |
Collapse
|
7
|
Willmeroth T, Wesselborg B, Kuske S. Implementation Outcomes and Indicators as a New Challenge in Health Services Research: A Systematic Scoping Review. INQUIRY : A JOURNAL OF MEDICAL CARE ORGANIZATION, PROVISION AND FINANCING 2019; 56:46958019861257. [PMID: 31347418 PMCID: PMC6661793 DOI: 10.1177/0046958019861257] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/13/2019] [Revised: 05/19/2019] [Accepted: 06/12/2019] [Indexed: 11/25/2022]
Abstract
The aim of this systematic scoping review was to identify and analyze indicators that address implementation quality or success in health care services and to deduce recommendations for further indicator development. This review was conducted according to the Joanna Briggs Manual and the PRISMA Statement. CINAHL, EMBASE, MEDLINE, and PsycINFO were searched. Studies or reviews published between August 2008 and 2018 that reported monitoring of the quality or the implementation success in health care services by using indicators based on continuous variables and proportion-based, ratio-based, standardized ratio-based, or rate-based variables or indices were included. The records were screened by title and abstract, and the full-text articles were also independently double-screened by 3 reviewers for eligibility. In total, 4376 records were identified that resulted in 10 eligible studies, including 67 implementation indicators. There was heterogeneity regarding the theoretical backgrounds, designs, objectives, settings, and implementation indicators among the publications. None of the indicators addressed the implementation outcomes of appropriateness or sustainability. Service implementation efficiency was identified as an additional outcome. Achieving consensus in framing implementation outcomes and indicators will be a new challenge in health services research. Considering the new debates regarding health care complexity, the further development of indicators based on complementary qualitative and quantitative approaches is needed.
Collapse
|
8
|
Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, Stanick C, Halko H, Powell BJ. An updated protocol for a systematic review of implementation-related measures. Syst Rev 2018; 7:66. [PMID: 29695295 PMCID: PMC5918558 DOI: 10.1186/s13643-018-0728-3] [Citation(s) in RCA: 40] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/20/2017] [Accepted: 04/11/2018] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Implementation science is the study of strategies used to integrate evidence-based practices into real-world settings (Eccles and Mittman, Implement Sci. 1(1):1, 2006). Central to the identification of replicable, feasible, and effective implementation strategies is the ability to assess the impact of contextual constructs and intervention characteristics that may influence implementation, but several measurement issues make this work quite difficult. For instance, it is unclear which constructs have no measures and which measures have any evidence of psychometric properties like reliability and validity. As part of a larger set of studies to advance implementation science measurement (Lewis et al., Implement Sci. 10:102, 2015), we will complete systematic reviews of measures that map onto the Consolidated Framework for Implementation Research (Damschroder et al., Implement Sci. 4:50, 2009) and the Implementation Outcomes Framework (Proctor et al., Adm Policy Ment Health. 38(2):65-76, 2011), the protocol for which is described in this manuscript. METHODS Our primary databases will be PubMed and Embase. Our search strings will be comprised of five levels: (1) the outcome or construct term; (2) terms for measure; (3) terms for evidence-based practice; (4) terms for implementation; and (5) terms for mental health. Two trained research specialists will independently review all titles and abstracts followed by full-text review for inclusion. The research specialists will then conduct measure-forward searches using the "cited by" function to identify all published empirical studies using each measure. The measure and associated publications will be compiled in a packet for data extraction. Data relevant to our Psychometric and Pragmatic Evidence Rating Scale (PAPERS) will be independently extracted and then rated using a worst score counts methodology reflecting "poor" to "excellent" evidence. DISCUSSION We will build a centralized, accessible, searchable repository through which researchers, practitioners, and other stakeholders can identify psychometrically and pragmatically strong measures of implementation contexts, processes, and outcomes. By facilitating the employment of psychometrically and pragmatically strong measures identified through this systematic review, the repository would enhance the cumulativeness, reproducibility, and applicability of research findings in the rapidly growing field of implementation science.
Collapse
Affiliation(s)
- Cara C. Lewis
- Kaiser Permanente Washington Health Research Institute, MacColl Center for Health Care Innovation, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101 USA
- Department of Psychological and Brain Sciences, Indiana University, 1101 E 10th Street, Bloomington, IN 47405 USA
- Department of Psychiatry and Behavioral Sciences, Harborview Medical Center, University of Washington, 325 9th Ave, Box 354946, Seattle, WA 98104 USA
| | - Kayne D. Mettert
- Kaiser Permanente Washington Health Research Institute, MacColl Center for Health Care Innovation, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101 USA
| | - Caitlin N. Dorsey
- Kaiser Permanente Washington Health Research Institute, MacColl Center for Health Care Innovation, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101 USA
| | - Ruben G. Martinez
- Psychology Department, Virginia Commonwealth University, 806 W. Franklin St, Box 842018, Richmond, VA 23284 USA
| | - Bryan J. Weiner
- Department of Global Health, University of Washington, 1510 San Juan Road, Box 357965, Seattle, WA 98195 USA
| | - Elspeth Nolen
- Department of Global Health, University of Washington, 1510 San Juan Road, Box 357965, Seattle, WA 98195 USA
| | - Cameo Stanick
- Hathaway-Sycamores Child and Family Services, 210 S DeLacey Ave, Suite 110, Pasadena, CA 91105-2074 USA
| | - Heather Halko
- Department of Psychology, University of Montana, 32 Campus Drive, Missoula, MT 59812 USA
| | - Byron J. Powell
- Department of Health Policy and Management, Gillings School of Global Public Health, University of North Carolina at Chapel Hill, 135 Dauer Drive, Chapel Hill, NC 27599 USA
| |
Collapse
|