Program theory evaluation: Logic analysis

Building a Logic Model to Foster Engagement and Learning Using the Case of a Province-Wide Multispecies Antimicrobial Use Monitoring System

Successfully designing and implementing a program is complex; it requires a reflexive balance between the available resources and the priorities of various stakeholders, both of which change over time. Logic models are theory-based evaluation approaches used to identify and address key challenges of a program. This article describes the process of building a logic model on advanced theories in complexity studies. The models aim to support a province-wide multispecies monitoring system of antimicrobial use (AMU), designed in collaboration with the animal health sector in Quebec (Canada). Based on a rigorous theoretical foundation, the logic model is built in three steps: (1) mapping, a narrative review of literature on similar programs in other jurisdictions; (2) framing, iterative consultations with project members to elaborate the logic model; (3) shaping, hypotheses based on the logic model. The model emerges from the reflexive balancing of current scientific knowledge and empirical insights to gather relevant information about stakeholders from interdisciplinary experts that led a 3-year consensus-building process within the community. Recognizing the challenge of unpacking theories for practical use, we illustrate how the process of an "open" logic model building could enable governance coordination in complex processes. Logic models are useful for evaluating public, private, and academic partnerships in One Health programs that characterize an adaptive governance process.

Practices for respecting the newborn's sleep-wake cycle: Interventional study in the neonatal intensive care unit

Objectives

Premature newborns are exposed to a great deal of over-stimulation, which can affect their cerebral development. For better sleep, certain practices should be recommended. The aim of this study was to evaluate the effect of a professional training program on the improvement of practices promoting respect for the newborn's sleep-wake cycle.

Materials and Methods

This was an interventional study with a longitudinal, single-group, and before-and-after design. The experimental design followed a three-stage time series: Eight months before, three months after, then eight months after intervention. It targeted a comprehensive sample of 66 professionals. It took place between October 2020 and March 2022 at the Neonatal Intensive Care Unit of the Hassan II University Hospital in Fez, Morocco. It was based on an observation grid and a self-administered questionnaire, validated and tested with a Cronbach's alpha reliability of 0.91.

Results

The light environment showed significant differences between the 1st and 3rd step (3.3% vs. 45.0%; P = 0.02; confidence interval [CI] = 13.644-10.456) for knowledge and (13.3% vs. 78.3%; P = 0.01; CI = 14.412-10.888) for practices; the noise environment showed a positive improvement between the 2nd and 3rd intervention (31.7% vs. 41.7%; P < 0.001; CI = 5.954-2.913) for knowledge and (65.0% vs. 73.3%; P < 0.001; CI = 3.597-1.236) for practices, with an average of (8.98 ± 0.30-28.15 ± 0.48; CI = 3.806-1.094) between the 1st and 3rd step. Practices surrounding sleep and wakefulness reported significant improvement between the three periods (14.35 ± 0.22 vs. 18.10 ± 0.35 vs. 19.90 ± 0.35; P P < 0.001; CI = 4.647-2.853) for sleep and (13.25 ± 0.48 vs. 22.27 ± 0.59; P < 0.001; CI = 10.563-7.471) for wakefulness with statistically significant correlations between knowledge and practices (0.426**) for sleep and (0.606**) for wakefulness.

Conclusion

The study demonstrated the positive impact of this sleep management and assessment program on the development of professional skills. Its implementation requires rigorous application of developmental support strategies for individualized care in neonatology.

A Settings and Systems Approach to Promoting the Health and Wellbeing of People with an Intellectual Disability

Research has shown that people with an intellectual disability have higher rates of certain preventable health conditions and a higher rate of avoidable mortality relative to the general population. With respect to health behaviours, they also have lower levels of physical activity and poorer nutrition. Despite the increased health needs, this population cohort is less likely to have the opportunity to participate in health promotion programs. The interventions that have been delivered are programmatic and individual in focus and have lacked a broader ecological and settings focus, which makes them very difficult to scale. Health promotion programs designed for the general population, such as lifestyle campaigns, rarely reach people with an intellectual disability. This increases the importance of ensuring that the settings in which they live and engage with are as health promoting as possible. Interventions have been particularly absent in the workplace for people with an intellectual disability. This paper highlights gaps in a settings-and-systems-based approach to promoting the health and wellbeing of people with an intellectual disability, particularly with respect to workplace health promotion. The paper concludes with recommendations for a systems approach that integrates approaches across multiple settings to better promote the health of this population cohort.

Evaluation of the outcomes of the Quebec provincial suicide prevention gatekeeper training on knowledge, recognition of attitudes, perceived self-efficacy, intention to help, and helping behaviors

INTRODUCTION

Gatekeeper (GK) training is a suicide prevention strategy in which community members learn to identify individuals at risk of suicide and refer them for appropriate help. Despite its widespread use, few studies have investigated its effects, including changes in helping behaviors.

AIMS

To assess the impact of GK training on participants' knowledge, recognition of the influence of attitudes, perceived self-efficacy, intention to help and helping behaviors, and to identify variables associated with GK behaviors.

METHODS

Mixed linear effects and forward stepwise logistic regressions were used to analyze data from 159 participants receiving the Quebec Provincial GK Training program offered by five different suicide prevention centers using pretest, posttest and 6-month follow-up questionnaires.

RESULTS

Participants' knowledge of the GK role and suicide prevention, intention to help, self-efficacy, knowledge of services, and recognition of the influence of attitudes significantly increased following training. Most changes decreased at follow-up but remained higher than at pretest. Lower levels of education and higher intention to help were significant predictors of engaging in helping behaviors in the first 6 months after receiving training.

CONCLUSIONS

The Quebec GK training appears to be effective in preparing participants for their role but does not appear to significantly increase helping behaviors.

A Critical Evaluation of International Agreements Towards a Revised Categorization for Transfrontier Conservation Areas (TFCAs)

Transfrontier Conservation Areas (TFCAs) are widely promoted as an international instrument to achieve certain conservation, cooperation and developmental goals, especially within the Southern African Development Community (SADC). In the SADC context, the status of TFCAs is categorized based on the extent to which international agreements have been signed. These agreements take different forms such as treaties, memorandums of understanding (MoUs), protocols and bilateral agreements. However, the efficacy of agreement-based approaches towards the categorization of TFCAs has been questioned because it does not acknowledge the implementation complexities of TFCAs and lacks a sound conceptual basis. This research evaluates the international TFCA agreements in SADC with a view to recommending a revised categorization. This is achieved by applying Theory of Change (ToC) to a sample of ten signed TFCAs agreements. The results show a lack of enforcement mechanisms, weak provision for implementation and poorly defined objectives. These weaknesses of agreement-based approaches can best be addressed by expanding the categorization of TFCAs to also include the extent of legislative and operational alignment. The revised categorization supports a more complete understanding of TFCA implementation.

Examining holistically the experiences of mentors in school-based programs: A logic analysis

The article presents a Logic Analysis of the Scottish MCR ("Motivation, Commitment and Resilience") Pathways school-based mentoring scheme. MCR Pathways provides vulnerable secondary school students with one-to-one support, helping them realize their full potential through education. The perceptions of 12 mentors were explored through interviews, thematically analyzed and mapped to derive the program's Theory of Change as regards the volunteers themselves. This model was then assessed against the evidence base yielded from prior studies. The evaluation highlighted a mismatch between mentors' outcome expectations and what they actually gained from the experience. Furthermore, some themes (e.g., being driven by community concerns) turned out to be more prominent in the context of this specific scheme than in the wider literature, as opposed to other ones (e.g., developing friendships). The study generates insights into the ways to attract and retain growing numbers of volunteers, as well as to advance scientific knowledge.

Factors Influencing the Effectiveness of Adopting Electronic Medical Record-Based Reporting Systems for Notifiable Disease Surveillance: A Quantitative Analysis

The coronavirus disease 2019 (COVID-19) pandemic has led to greater attention being given to infectious disease surveillance systems and their notification functionalities. Although numerous studies have explored the benefits of integrating functionalities with electronic medical record (EMR) systems, empirical studies on the topic are rare. The current study assessed which factors influence the effectiveness of EMR-based reporting systems (EMR-RSs) for notifiable disease surveillance. This study interviewed staff from hospitals with a coverage that represented 51.39% of the notifiable disease reporting volume in Taiwan. Exact logistic regression was employed to determine which factors influenced the effectiveness of Taiwan's EMR-RS. The results revealed that the influential factors included hospitals' early participation in the EMR-RS project, frequent consultation with the information technology (IT) provider of the Taiwan Centers for Disease Control (TWCDC), and retrieval of data from at least one internal database. They also revealed that using an EMR-RS resulted in more timely, accurate, and convenient reporting in hospitals. In addition, developing by an internal IT unit instead of outsourcing EMR-RS development led to more accurate and convenient reporting. Automatically loading the required data enhanced the convenience, and designing input fields that may be unavailable in current databases to enable physicians to add data to legacy databases also boosted effectiveness of the reporting system.

Implementation process evaluation of an ergonomic train the trainer program: How to learn from mechanisms and the temporal structure of processes?

In implementation process evaluation, the analysis of the temporal structure of processes is key for understanding the successive interactions between the flow of practitioners' actions and evolving workplace reactions and context. However, capturing the temporal structure of processes in data analysis is a methodological challenge, and available literature to overcome this challenge is scarce, especially for workplace ergonomic interventions. The aim of this paper was to perform an implementation process evaluation of an ergonomic train the trainer program taking into account the temporal structure of processes. We provided a method for qualitative data analysis based on a three-stage strategy: 1) producing the timeline of the implemented intervention, 2) identifying influential factors, 3) identifying determinant mechanisms (sequence of influential factors that intervened in the implementation process). This method allowed us to identify six determinant mechanisms positioned on the timeline of the intervention. Obstacles and levers were identified as a sequence of interrelated causes and consequences rather than isolated factors. We recommended success strategies for practitioners, while also shedding light on how organizations can better be prepared to undertake the intervention and their required actions to attain targeted intervention objectives.

A realist review of the home care literature and its blind spots

RATIONALE, AIMS AND OBJECTIVES

There is a large body of literature from all over the world that describes, analyzes, or evaluates home care models and interventions. The present article aims to identify the practical lessons that can be gained from a systematic examination of that literature.

METHOD

We conducted a three-step sequential search process from which 113 documents were selected. That corpus was then narratively analysed according to a realist review approach.

RESULTS

A first level of observation is that there are multiple blind spots in the existing literature on home care. The definition and delimitation of what constitutes home care services is generally under-discussed. In the same way, the composition of the basket of care provided and its fit with the need of recipients is under-addressed. Finally, the literature relies heavily on RCTs whose practical contribution to decisions or policy is disputable. At a second level, our analysis suggests that three mechanisms (system integration, case management and relational continuity) are core characteristics of home care models' effectiveness.

CONCLUSION

We conclude by providing advice for supporting the design and implementation of stronger home care delivery systems. Our analysis suggests that doing so implies a series of sequential steps: identify what system-level goals the model should achieve and which populations it should serve; identify what type of services are likely to achieve those goals in order to establish a basket of services; and finally, identify the best ways and specific means to effectively and efficiently provide those services. Those same steps can also support ex-post evaluations of existing home care systems.

Identifying key risks to the performance of privately protected areas (PPAs) through theory of change (ToC)

Privately protected areas (PPAs) are internationally considered to be important policy implementation instruments to augment and strengthen protected area networks. However, there has been limited reflection on the performance of PPAs over time. This paper aims to identify key risks to the performance of PPAs as policy implementation instruments through the application of Theory of Change (ToC). Identifying and understanding these risks are important to allow for the evaluation and monitoring of PPA performance. The ToC method was applied to a specific PPA policy instrument namely, private nature reserves (PNRs) in the South African context. The research results produced 29 key assumptions translated into 29 key risks. These risk are critically discussed against existing South African and international literature. To test and refine the risks further it is recommended that they be applied to PPA case studies in different contexts.

Managers do it their way: How managers act in a decentralised healthcare services provider organisation - a mixed methods study

Purpose: Decentralisation is considered a way to get managers more committed and more prone to respond to local needs. This study analyses how managers perceive a decentralised management model within a large public healthcare delivery organisation in Sweden. Design/methodology/approach: A programme theory evaluation was performed applying direct content analysis to in-depth interviews with healthcare managers. Balance score card data were used in a blinded comparative content analysis to explore relations between performance and how the delegated authority was perceived and used by the managers. Findings: Managers' perceptions of the decentralised management model supported its intentions to enable the front-line to make decisions to better meet customer needs and flexibly adapt to local conditions. The managers appreciated and used their delegated authority. Central policies and control on human resources and investments were accepted as those are to the benefit of the whole organisation. Leadership development and organisation-wide improvement programmes were of support. Units showing high organisational performance had proactive managers, although differences in manager perceptions across units were small. Originality: This, one of the first of its kind, study of a decentralisation in service delivery organisation shows a congruence between the rationale of a management model, the managers' perceptions of the authority and accountability as well as management practises. These observations stemming from a large public primary and community healthcare organisation has not, to our knowledge, been reported and provide research-informed guidance on decentralisation as one strategy for resolving challenges in healthcare service delivery organisations.

Designing centralized waiting lists for attachment to a primary care provider: Considerations from a logic analysis

Access to a regular primary care provider is essential to quality care. In Canada, where 15 % of patients are unattached (i.e., without a regular provider), centralized waiting lists (CWLs) help attach patients to a primary care provider (family physician or nurse practitioner). Previous studies reveal mechanisms needed for CWLs to work, but focus mostly on CWLs for specialized health care. We aim to better understand how to design CWLs for unattached patients in primary care. In this study, a logic analysis compares empirical evidence from a qualitative case study of CWLs for unattached patients in seven Canadian provinces to programme theory derived from a realist review on CWLs. Data is analyzed using context-intervention-mechanism-outcome configurations. Results identify mechanisms involved in three components of CWL design: patient registration, patient prioritization, and patient assignment to a provider for attachment. CWL programme theory is revised to integrate mechanisms specific to primary care, where patients, rather than referring providers, are responsible for registering on the CWL, where prioritization must consider a broad range of conditions and characteristics, and where long-term acceptability of attachment is important. The study provides new insight into mechanisms that enable CWLs for unattached patients to work.

Do mentoring programmes influence women's careers in the health and medical research sector? A mixed-methods evaluation of Australia's Franklin Women Mentoring Programme

OBJECTIVES

It is known that women are under-represented in senior positions within the health and medical research sector. The Franklin Women Mentoring Programme (Programme) is a professionally facilitated, cross-organisational initiative designed to support career development for mid-career women. The objective of this study was to evaluate Programme outcomes reported by participants 12 months following its formal conclusion.

DESIGN

Explanatory sequential mixed-methods study design using a cross-sectional survey and semi-structured interviews.

SETTING

Health and medical research institutes in Sydney, Australia.

PARTICIPANTS

Health and medical researchers from the 2018 Programme.

PRIMARY AND SECONDARY OUTCOME MEASURES

Changes in knowledge, skills, behaviours and research metrics directly attributed to Programme participation.

RESULTS

A total of 50 mentors and mentees participated in the cross-sectional survey (68% of the total cohort) and 14 mentors and mentees participated in the interviews. All reported changes to their knowledge, skills, behaviours and research metrics which were directly attributed to participation in the Programme. This included changes in knowledge and skills to be more inclusive (96% mentees, 83% mentors), resilience (88% mentees, 67% mentors), ability to have difficult workplace conversations (88% mentees, 71% mentors) and improvements in supervisory and team management (82% mentees, 75% mentors) skills. Positive impacts on promotions and grant opportunities were also reported. All evaluation participants believed this Programme was a worthwhile initiative for their workplaces to invest in.

CONCLUSION

Participation in this cross-organisational, professionally facilitated, structured mentoring programme has led to positive outcomes for mentees, as well as mentors. Reported outcomes indicate the Programme is meeting its aims to support the career development of mid-career women in health and medical research, while facilitating a more inclusive workforce.

Youth and parent perceptions on participating in specialized multidisciplinary pain rehabilitation options: A qualitative timeline effect analysis

Background: Little is known about how the specialized treatment journey is perceived by youth with pain-related disability and their parents. Aims: Describe and compare the treatment effects and outcomes as perceived by youth and their parents enrolled in intensive interdisciplinary pain treatment (IIPT) or multimodal treatment (MMT). Methods: Eleven IIPT youth and five parents and three MMT youth and five parents were recruited. All were asked to complete a treatment journey timeline, followed by separately conducted semistructured interviews. Transcribed interviews were analyzed using reflective thematic analysis. Results: The main themes spanned the treatment trajectory. All participants described similar initial struggles (Theme 1). Positive and negative treatment effects associated with acquisitions and disruptions (Theme 2), and outcomes post-discharge related to supports and realities (Theme 3) emerged. Knowledge, skills, and support acquisition during treatment and feeling empowered and confident to self-manage postdischarge were identified as IIPT benefits. However, the change effort and life disruptions required and the difficulty transitioning to real life postprogram were acknowledged as detrimental IIPT impacts. Continuing with life as usual and maintaining supports in daily contexts (e.g., school personnel, friends) were reported MMT benefits. However, the challenges of managing pain, treatment adherence within the competing demands of daily realities, and the lack of support to integrate strategies were emphasized as detrimental MMT impacts. Conclusions: Detailed impacts of two specialized multidisciplinary pain rehabilitation interventions on the lives of youth with pain-related disability and their parents are described. The treatments benefits and previously unexplored detrimental effects are unveiled.

Logic models for the Therapeutic Return-to-Work Program as adapted for common mental disorders: A guide for health professionals

BACKGROUND

Workplace interventions are recommended for workers with common mental disorders, but knowledge of their action mechanisms and operationalization remains limited. The Therapeutic Return-to-Work Program, developed for workers with musculoskeletal disorders, is recommended for common mental disorders.

OBJECTIVE

Our objective was to adapt this program's logic models to common mental disorders.

METHODS

A program logic analysis was conducted using a literature review and a two-phase group consensus method. We submitted a preliminary adapted version of the program's logic models and two questionnaires to health professional experts who participated in two group sessions, ultimately to produce the final version of the models.

RESULTS

We consulted 86 publications. The health professional experts (N = 7) had overall mean agreement scores of respectively 4.10/5 and 3.89/5 for questions on the program's theoretical and operational models. The final version of the logic models adapted for common mental disorders included four specific and 15 intermediate objectives, three main components, one optional component, four key processes, and 44 tasks.

CONCLUSION

The adapted logic models for the Therapeutic Return-to-Work Program show the relevance of the original objectives and components for common mental disorders. The next step will involve evaluating its feasibility with other stakeholders (insurers, employers, unions, workers).

Theoretical analysis of policies to improve the recruitment and retention of rural physicians

The lack of success in resolving the shortage of rural physicians in Organisation for Economic Cooperation and Development countries has been attributed to the weakness of implemented policies. This research examines the theoretical plausibility of policies to improve the recruitment and retention of rural physicians, first, by modelling the policies; and then, by describing how they might achieve their intended outcome based on a theoretical analysis. A theory-driven method relying on published research and expert analysis is used. A conceptual model is created to represent the policies and their underlying assumptions. Then, the functional mechanism of the policies is defined. This research demonstrates that financial, educational and tailored interventions might improve rural physician recruitment and retention, but that regulatory interventions are unlikely to do this. The majority of the policies implemented in Organisation for Economic Cooperation and Development countries are therefore theoretically plausible.

Using collaborative logic analysis evaluation to test the program theory of an intensive interdisciplinary pain treatment for youth with pain‐related disability

Intensive interdisciplinary pain treatment (IIPT) involves multiple stakeholders. Mapping the program components to its anticipated outcomes (ie, its theory) can be difficult and requires stakeholder engagement. Evidence is lacking, however, on how best to engage them. Logic analysis, a theory‐based evaluation, that tests the coherence of a program theory using scientific evidence and experiential knowledge may hold some promise. Its use is rare in pediatric pain interventions, and few methodological details are available. This article provides a description of a collaborative logic analysis methodology used to test the theoretical plausibility of an IIPT designed for youth with pain‐related disability. A 3‐step direct logic analysis process was used. A 13‐member expert panel, composed of clinicians, teachers, managers, youth with pain‐related disability, and their parents, were engaged in each step. First, a logic model was constructed through document analysis, expert panel surveys, and focus‐group discussions. Then, a scoping review, focused on pediatric self‐management, building self‐efficacy, and fostering participation, helped create a conceptual framework. An examination of the logic model against the conceptual framework by the expert panel followed, and recommendations were formulated. Overall, the collaborative logic analysis process helped raiseawareness of clinicians’ assumptions about the program causal mechanisms, identified program components most valued by youth and their parents, recognized the program features supported by scientific and experiential knowledge, detected gaps, and highlighted emerging trends. In addition to providing a consumer‐focused program evaluation option, collaborative logic analysis methodology holds promise as a strategy to engage stakeholders and to translate pediatric pain rehabilitation evaluation research knowledge to key stakeholders.

Evaluating complex interventions in real context: Logic analysis of a case management program for frequent users of healthcare services

Case management programs for frequent users of healthcare services are complex interventions which implementation and application are challenging to evaluate. The aim of this article was to conduct a logic analysis to evaluate a case management program for frequent users of healthcare services. The study proceeded in three phases: 1) establishing causal links between the program's components by the construction of a logic model, 2) developing an integrated framework from a realistic synthesis, and 3) making a new reading of the case management program in regard of the integrated framework. The study demonstrated, on one hand, strengths and weaknesses of the actual case management program, and, on the other hand, how logic analysis can create a constructive dialogue between theory and practice. The evaluative process with decision-makers, clinicians and patients has helped to make connexions between theory, practice, experience and services organization.

Embedding value-for-money in practice: A case study of a health pooled fund programme implemented in conflict-affected South Sudan

In recent times, there has been an increasing drive to demonstrate value for money (VfM) for investments made in public health globally. However, there is paucity of information on practical insights and best practices that have helped implementing organisations to successfully embed VfM in practice for programming and evaluation. In this article, we discuss strengths and weaknesses of approaches that been used and insights on best practices to manage for, demonstrate, and compare VfM, using a health pooled fund programme implemented in conflict-affected South Sudan as case study supported by evidence reported in the literature while critiquing adequacy of the available approaches in this setting. An expanded and iterative process framework to guide VfM embedding for health programming and evaluation is then proposed. In doing so, this article provides a very relevant one-stop source for critical insight into how to embed VfM in practice. Uptake and scale-up of the proposed framework can be essential in improving VfM and aid effectiveness which will ultimately contribute to progress towards achieving the Sustainable Development Goals by 2030.

Evaluation of a community of practice for speech-language pathologists in aphasia rehabilitation: a logic analysis

BACKGROUND

Aphasia is a communication disorder affecting participation. Although there are evidence-based practice recommendations about participation and aphasia rehabilitation, it may be challenging for speech-language pathologists to ensure that rehabilitation activities have an impact on the person's participation, in part due to time limitations. Participation remains limited after rehabilitation for persons who have aphasia. Communities of practice (CoPs) are a collaborative knowledge transfer strategy that can be used for evidence-based practice implementation. The aim of this study was to describe the components and evaluate a CoP for speech-language pathologists about participation and aphasia rehabilitation.

METHODS

Logic analysis was used to determine the adequacy between resources, implemented activities, outputs and short-term outcomes of the CoP. Qualitative and quantitative descriptive data were collected through observation and participants' logbooks. Outputs and outcomes of the CoP were revealed through thematic analysis and interpretation of descriptive statistics.

RESULTS

Resources including CoP design and educational aims, human and material resources were combined to create various web-based, online and offline activities. Participants invested more time per week than expected in the CoP, shared and created clinical tools and appreciated the array of suggested activities. Participant engagement allowed them to reflect, interact and collaborate with each other. All 13 participants reported they acquired knowledge about clinical tools and 12 mentioned they reflected on their practice. While the CoP was ongoing, six participants noticed evidence-practice gaps, seven prepared to change their practice, and three changed their practice towards including more participation-based considerations.

CONCLUSIONS

This study showed that speech-language pathologists can include more participation-based approaches in aphasia rehabilitation as a result of participating in a time-bound, web-based CoP.

A Core Components Framework for Evaluating Implementation of Competency-Based Medical Education Programs

PURPOSE

The rapid adoption of competency-based medical education (CBME) provides an unprecedented opportunity to study implementation. Examining "fidelity of implementation"-that is, whether CBME is being implemented as intended-is hampered, however, by the lack of a common framework. This article details the development of such a framework.

METHOD

A two-step method was used. First, a perspective indicating how CBME is intended to bring about change was described. Accordingly, core components were identified. Drawing from the literature, the core components were organized into a draft framework. Using a modified Delphi approach, the second step examined consensus amongst an international group of experts in CBME.

RESULTS

Two different viewpoints describing how a CBME program can bring about change were found: production and reform. Because the reform model was most consistent with the characterization of CBME as a transformative innovation, this perspective was used to create a draft framework. Following the Delphi process, five core components of CBME curricula were identified: outcome competencies, sequenced progression, tailored learning experiences, competency-focused instruction, and programmatic assessment. With some modification in wording, consensus emerged amongst the panel of international experts.

CONCLUSIONS

Typically, implementation evaluation relies on the creation of a specific checklist of practices. Given the ongoing evolution and complexity of CBME, this work, however, focused on identifying core components. Consistent with recent developments in program evaluation, where implementation is described as a developmental trajectory toward fidelity, identifying core components is presented as a fundamental first step toward gaining a more sophisticated understanding of implementation.

Unravelling how and why the Antiretroviral Adherence Club Intervention works (or not) in a public health facility: A realist explanatory theory-building case study

Background

Although empirical evidence suggests that the adherence club model is more effective in retaining people living with HIV in antiretroviral treatment care and sustaining medication adherence compared to standard clinic care, it is poorly understood exactly how and why this works. In this paper, we examined and made explicit how, why and for whom the adherence club model works at a public health facility in South Africa.

Methods

We applied an explanatory theory-building case study approach to examine the validity of an initial programme theory developed a priori. We collected data using a retrospective cohort quantitative design to describe the suppressive adherence and retention in care behaviours of patients on ART using Kaplan-Meier methods. In conjunction, we employed an explanatory qualitative study design using non-participant observations and realist interviews to gain insights into the important mechanisms activated by the adherence club intervention and the relevant contextual conditions that trigger the different mechanisms to cause the observed behaviours. We applied the retroduction logic to configure the intervention-context-actor-mechanism-outcome map to formulate generative theories.

Results

A modified programme theory involving targeted care for clinically stable adult patients (18 years+) receiving antiretroviral therapy was obtained. Targeted care involved receiving quick, uninterrupted supply of antiretroviral medication (with reduced clinic visit frequencies), health talks and counselling, immediate access to a clinician when required and guided by club rules and regulations within the context of adequate resources, and convenient (size and position) space and proper preparation by the club team. When grouped for targeted care, patients feel nudged, their self-efficacy is improved and they become motivated to adhere to their medication and remain in continuous care.

Conclusion

This finding has implications for understanding how, why and under what health system conditions the adherence club intervention works to improve its rollout in other contexts.

Evaluation of clinical quality improvement interventions: feasibility of an integrated approach

BACKGROUND

Cardiovascular diseases (CVD) are the largest cause of death and disability in Australia. Australian national guidelines for the primary prevention of CVD recommend that all adults without CVD and aged 45 years or more are screened for their absolute risk of CVD every 2 years. Despite the compelling evidence to address CVD risk, treatment gaps remain and evidence suggests that much of the shortcomings are attributed to the performance of primary care practices. To address this issue, a quality improvement initiative is being implemented in a large urban multidisciplinary primary care practice in the South West region of Victoria, Australia. The key outcome of this intervention will be to increase the use and acceptability of CVD risk assessment guidelines. To ensure the intervention is tracking toward its objectives, a robust monitoring and evaluation framework was established.

METHOD/DESIGN

A novel framework that assimilates key traditional and theory-driven evaluation practices was developed to assess the impact of the intervention. The framework approach is termed the integrated model of evaluation (IMoE). Researchers and stakeholders convened several times to discuss and develop the evaluation protocol and align it with the quality intervention. The main objective here is to explore the feasibility of an integrated approach to evaluating clinical quality improvement interventions. The sub-objectives are to test the alignment of the IMoE to clinical quality improvement projects and its ability to derive findings to the satisfaction of stakeholders. The design and establishment of the evaluation approach is discussed in further detail in this article.

DISCUSSION

The novel feature of the IMoE is its emphasis on tracking 'change' in practices that lead to quality improvement. This emphasis suits the quality improvement theme of this initiative as identification of change elements and explanation behind change is necessary to sustain and promote quality improvement. The other principle behind development of this model, which emphasises practicality in implementation, is to ensure stakeholders gain greatest value from the commissioning of program evaluation. By incorporating practical components and leaving out esoteric concepts, this approach ensures evaluation can be undertaken in realistic timeframes.

ETHICS APPROVAL

The quality improvement intervention and evaluation framework received approval from the Deakin University Human Research Ethics Committee (Approval Number: 2017-313).

Centralized Waiting Lists for Unattached Patients in Primary Care: Learning from an Intervention Implemented in Seven Canadian Provinces

Introduction

Centralized waiting lists (CWLs) are one solution to reduce the problematic number of patients without a regular primary care provider. This article describes different models of CWLs for unattached patients implemented in seven Canadian provinces and identifies common issues in the implementation of these CWLs.

Methods

Logic models of each province's intervention were built after a grey literature review, 42 semi-structured interviews and a validation process with key stakeholders were performed.

Results

Our analysis across provinces showed variability and common features in the design of CWLs such as same main objective to attach patients to a primary care provider; implementation as a province-wide program with the exception of British Columbia; management at a regional level in most provinces; voluntary participation for providers except in two provinces where it was mandatory for providers to attach CWL patients; fairly similar registration process across the provinces; some forms of prioritization of patients either using simple criteria or assessing for vulnerability was performed in most provinces except New Brunswick.

Conclusion

Despite their differences in design, CWLs implemented in seven Canadian provinces face common issues and challenges regarding provider capacity to address the demand for attachment, barriers to the attachment of more vulnerable and complex patients as well as non-standardized approaches to evaluating their effectiveness. Sharing experiences across provinces as CWLs were being implemented would have fostered learning and could have helped avoid facing similar challenges.

Effectiveness of the settings-based intervention Shaping the Social on preventing dropout from vocational education: a Danish non-randomized controlled trial

BACKGROUND

Lack of formal education is an important social determinant of health inequality and represents a public health problem. School dropout is particularly common in vocational education; however few prevention programs targeting dropout in the vocational school setting have been evaluated. The purpose of the present study was to test the effect on school dropout of a settings-based intervention program (named Shaping the Social) that targeted the school organization in order to create social and supportive learning environments.

METHODS

A non-randomized controlled design including four large intervention schools and six matched-control schools was used. The target population was students in technical and agricultural vocational education, which is provided to students from age 16. Students were enrolled at school start. Register-based data (n = 10,190) was used to assess the effect on school dropout during a 2-year period. Odds ratios (OR) and 95% confidence intervals (CI) were calculated in logistic regression models, adjusting for age, sex, ethnicity, parental income, prior school dropout and type of basic course. Student survey (n = 2396) at 10-week follow-up was used to examine wellbeing at school (four subscales: school connectedness, student support, teacher relatedness, and valuing the profession) which was the hypothesized proximal intervention effect. As a secondary aim, we examined how the student wellbeing factors were associated with school dropout, independently of the intervention, and we explored whether the student wellbeing factors were potential mediators.

RESULTS

The present study showed an intervention effect on school dropout with dropout rates lower in intervention schools (36%) than control schools (40%) (OR = 0.86, 95% CI: 0.74, 0.99). We had no attrition on the dropout outcome. School connectedness mediated the intervention effect; no significant mediation effects were found for student support, teacher relatedness, and valuing the profession. Independently of the intervention, each student wellbeing factor prevented dropout.

CONCLUSIONS

Findings from this study suggest that a comprehensive, multicomponent school-based intervention could prevent dropout from vocational education by promoting school connectedness; nevertheless, the dropout rate remained high. Our results point to the need to explore how to further improve the wellbeing at school among young people in vocational education.

TRIALS REGISTRATION

ISRCTN, ISRCTN57822968 . Registered 16 January 2013 (retrospective registered).

Analysing mHealth usage logs in RCTs: Explaining participants' interactions with type 2 diabetes self-management tools

Background

The Introduction of mobile health (mHealth) devices to health intervention studies challenges us as researchers to adapt how we analyse the impact of these technologies. For interventions involving chronic illness self-management, we must consider changes in behaviour in addition to changes in health. Fortunately, these mHealth technologies can record participants’ interactions via usage-logs during research interventions.

Objective

The objective of this paper is to demonstrate the potential of analysing mHealth usage-logs by presenting an in-depth analysis as a preliminary study for using behavioural theories to contextualize the user-recorded results of mHealth intervention studies. We use the logs collected by persons with type 2 diabetes during a randomized controlled trial (RCT) as a use-case.

Methods

The Few Touch Application was tested in a year-long intervention, which allowed participants to register and review their blood glucose, diet and physical activity, goals, and access general disease information. Usage-logs, i.e. logged interactions with the mHealth devices, were collected from participants (n = 101) in the intervention groups. HbA1c was collected (baseline, 4- and 12-months). Usage logs were categorized into registrations or navigations.

Results

There were n = 29 non-mHealth users, n = 11 short-term users and n = 61 long-term users. Non-mHealth users increased (+0.33%) while Long-term users reduced their HbA1c (-0.86%), which was significantly different (P = .021). Long-term users significantly decreased their usage over the year (P < .001). K-means clustering revealed two clusters: one dominated by diet/exercise interactions (n = 16), and one dominated by BG interactions and navigations in general (n = 40). The only significant difference between these two clusters was that the first cluster spent more time on the goals functionalities than the second (P < .001).

Conclusion

By comparing participants based upon their usage-logs, we were able to discern differences in HbA1c as well as usage patterns. This approach demonstrates the potential of analysing usage-logs to better understand how participants engage during mHealth intervention studies.

Public project success as seen in a broad perspective.: Lessons from a meta-evaluation of 20 infrastructure projects in Norway

Infrastructure projects in developed countries are rarely evaluated ex-post. Despite their number and scope, our knowledge about their various impacts is surprisingly limited. The paper argues that such projects must be assessed in a broad perspective that includes both operational, tactical and strategic aspects, and unintended as well as intended effects. A generic six-criteria evaluation framework is suggested, inspired by a framework frequently used to evaluate development assistance projects. It is tested on 20 Norwegian projects from various sectors (transport, defence, ICT, buildings). The results indicate that the majority of projects were successful, especially in operational terms, possibly because they underwent external quality assurance up-front. It is argued that applying this type of standardized framework provides a good basis for comparison and learning across sectors. It is suggested that evaluations should be conducted with the aim of promoting accountability, building knowledge about infrastructure projects, and continuously improve the tools, methods and governance arrangements used in the front-end of project development.

Studies evaluating of health interventions at schools: an integrative literature review

OBJECTIVE

to identify and analyze the available evidence on the strategies used in the studies evaluating health interventions at school.

METHOD

this is an integrative review searching in LILACS, CINAHL, CUIDEN, ScienceDirect, and PubMed. From the pre-defined inclusion and exclusion criteria, there were 121 articles chosen to compose the sample.

RESULTS

english studies (97.5%), with a quantitative approach (80.2%), related to the interventions carried out in the Region of the Americas (54.6%) and the European Region (23.1%) predominated. For the most part, they are interventions as programs (70.2%), interested in evaluating results (73.5%) from the value judgment (83.4%). Prevalence of interventions focused on efficacy, effects or impact, and activities carried out on interventions were focused on physical activity, healthy eating, sexual and reproductive health, mental health, and use of tobacco, alcohol, and other drugs. They are worked through activities of clinical monitoring, health promotion and disease prevention.

CONCLUSION

the evidence indicates that the evaluations of health interventions in the school focus the results produced in programs through the judgment of value. The topics most addressed were healthy eating, physical activity, prevention of alcohol and other drugs, among others.

Analytical typology of multiprofessional primary care models

Background

There is only limited evidence to support care redefinition and role optimization processes needed for scaling up of a stronger primary care capacity.

Methods

Data collection was based on a keyword search in MEDLINE, EMBASE and CINAHL databases. Three thousand, two hundred and twenty-nine documents were identified, 1851 met our inclusion criteria, 71 were retained for full-text assessment and 52 included in the final selection. The analysis process was done in four steps. In the end, the elements that were identified as particularly central to the process of transforming primary care provision were used as the basis of two typologies.

Results

The first typology is based on two structural dimensions that characterize promising multiprofessional primary care teams. The first is the degree to which the division of tasks in the team was formalized. The second dimension is the centrality and autonomy of nurses in the care model. The second typology offers a refined definition of comprehensiveness of care and its relationship with the optimization of professional roles.

Conclusions

The literature we analyzed suggests there are several plausible avenues for coherently articulating the relationships between patients, professionals, and care pathways. The expertise, preferences, and numbers of available human resources will determine the plausibility that a model will be a coherent response that is appropriate to the needs and environmental constraints (funding models, insurance, etc.). The typologies developed can help assess existing care models analytically or evaluatively and to propose, prospectively, some optimal operational parameters for primary care provision.

Program theory within policy-initiated evaluations: the Norwegian low-income family study

Using as an example a project where the Norwegian Labor and Welfare Directorate developed a comprehensive model for the follow-up of low-income families, this article demonstrates the process of developing a program theory for policy-initiated interventions. The data consist of interviews with program developers, political documents from early stages, and observations of the program's development. The results demonstrate that, although research inspired the program developers, the program was also the outcome of policy priorities, experiences from earlier projects, and input from the practice field. Multiple sources contributed to its relevance for the practice field, however, increasing its complexity. The program includes several intervention levels and follow-up areas and partially builds on elements found to be important across interventions. Although a program theory can be difficult to conceptualize within policy-initiated interventions, it is important to articulate it prior to evaluation and, if necessary, reassess it when data have been analyzed.

Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences

This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.

Whole Medical Systems versus the System of Conventional Biomedicine: A Critical, Narrative Review of Similarities, Differences, and Factors That Promote the Integration Process

BACKGROUND

There is an increasing need for a worldwide professional integration of conventional medicine and traditional/complementary whole medical systems (WMSs). However, the integration is perceived by conventional medicine as problematic or unacceptable, because of a supposed lack of evidence for specific effects of WMSs therapies and supposed prescientific or unscientific paradigms of WMSs.

OBJECTIVES

To review the literature on the features of WMSs, similarities and differences between conventional medicine and WMSs, and scientific and clinical practice issues that should be dealt with in order to promote the integration process.

METHODS

A critical, narrative review of the literature on six WMSs.

RESULTS AND CONCLUSIONS

Key factors for the integration of WMSs and conventional medicine are as follows: legal frameworks, quality standards, high-quality research on safety and efficacy of WMS interventions, infrastructure, and financial resources. For scientific assessment of WMSs, there are unresolved ontological, epistemological, and methodological issues and issues of diagnostics, therapy delivery, and outcome assessment in clinical practice. Future research not only should be directed at quality assurance and generating the necessary data on safety and efficacy/effectiveness but also should address more fundamental (ontological, epistemological, and methodological) issues, in order to overcome the differences between WMSs and conventional medicine.

A hybrid method for evaluating enterprise architecture implementation

Enterprise Architecture (EA) implementation evaluation provides a set of methods and practices for evaluating the EA implementation artefacts within an EA implementation project. There are insufficient practices in existing EA evaluation models in terms of considering all EA functions and processes, using structured methods in developing EA implementation, employing matured practices, and using appropriate metrics to achieve proper evaluation. The aim of this research is to develop a hybrid evaluation method that supports achieving the objectives of EA implementation. To attain this aim, the first step is to identify EA implementation evaluation practices. To this end, a Systematic Literature Review (SLR) was conducted. Second, the proposed hybrid method was developed based on the foundation and information extracted from the SLR, semi-structured interviews with EA practitioners, program theory evaluation and Information Systems (ISs) evaluation. Finally, the proposed method was validated by means of a case study and expert reviews. This research provides a suitable foundation for researchers who wish to extend and continue this research topic with further analysis and exploration, and for practitioners who would like to employ an effective and lightweight evaluation method for EA projects.

Failing better: The stochastic art of evaluating community-led environmental action programs

This article provides insights into the evaluation of a government-funded action for climate change program. The UK-based program aimed to reduce CO2 emissions and encourage behavioral change through community-led environmental projects. It, thus, employed six community development facilitators, with expertise in environmental issues. These facilitators supported and learnt from 18 community groups over an 18-month period. The paper explores the narratives of the six professional facilitators. These facilitators discuss their experiences of supporting community groups. They also explain their contribution to the wider evaluation of the community-led projects. This paper reflects on the facilitator experience of the program's outcome-led evaluation process. In turn, it also explores how the groups they supported experienced the process. The facilitator's narratives reveal that often community-group objectives did not align with predefined outcomes established through theory of change or logic model methodologies, which had been devised in attempt to align to program funder aims. Assisting community action emerges in this inquiry as a stochastic art that requires funder and facilitator willingness to experiment and openness to the possibilities of learning from failure. Drawing on in-depth accounts, the article illustrates that a reflexive, interpretive evaluation approach can enhance learning opportunities and provides funders with more trustworthy representations of community-led initiatives. Yet, it also addresses why such an approach remains marginal within policy circles.

A comparative analysis of centralized waiting lists for patients without a primary care provider implemented in six Canadian provinces: study protocol

BACKGROUND

Having a regular primary care provider (i.e., family physician or nurse practitioner) is widely considered to be a prerequisite for obtaining healthcare that is timely, accessible, continuous, comprehensive, and well-coordinated with other parts of the healthcare system. Yet, 4.6 million Canadians, approximately 15% of Canada's population, are unattached; that is, they do not have a regular primary care provider. To address the critical need for attachment, especially for more vulnerable patients, six Canadian provinces have implemented centralized waiting lists for unattached patients. These waiting lists centralize unattached patients' requests for a primary care provider in a given territory and match patients with providers. From the little information we have on each province's centralized waiting list, we know the way they work varies significantly from province to province. The main objective of this study is to compare the different models of centralized waiting lists for unattached patients implemented in six provinces of Canada to each other and to available scientific knowledge to make recommendations on ways to improve their design in an effort to increase attachment of patients to a primary care provider.

METHODS

A logic analysis approach developed in three steps will be used. Step 1: build logic models that describe each province's centralized waiting list through interviews with key stakeholders in each province; step 2: develop a conceptual framework, separate from the provincially informed logic models, that identifies key characteristics of centralized waiting lists for unattached patients and factors influencing their implementation through a literature review and interviews with experts; step 3: compare the logic models to the conceptual framework to make recommendations to improve centralized waiting lists in different provinces during a pan Canadian face-to-face exchange with decision-makers, clinicians and researchers.

DISCUSSION

This study is based on an inter-provincial learning exchange approach where we propose to compare centralized waiting lists and analyze variations in strategies used to increase attachment to a regular primary care provider. Fostering inter-provincial healthcare systems connectivity to improve centralized waiting lists' practices across Canada can lever attachment to a regular provider for timely access to continuous, comprehensive and coordinated healthcare for all Canadians and particular for those who are vulnerable.

Strategic analysis of tuberculosis prevention and control actions in Brazil and Ethiopia: one size fits all?

OBJECTIVES

This study aimed at conducting a strategic analysis of Tuberculosis prevention and control actions in Brazil and Ethiopia, looking at the potential of directly observed treatment short-course strategy (DOTS) and community DOTS in both countries.

METHODS

Literature review was conducted using PubMed, Medline-Ovid, EMBASE, and SCIELO databases. The reviewed terms were Tuberculosis, prevention and control and Brazil (or Brasil) or Ethiopia (or Etiopia). Study's eligibility included article's title or abstract in English or Portuguese and comprised the following Tuberculosis policy components: management; care; communication, and social mobilization; training and professional development; epidemiological surveillance, and monitoring and evaluation. The study identified, compared, and analyzed the challenges and recommendations reported in the literature.

RESULTS

Although DOTS was not able to address all the difficulties regarding Tuberculosis control and prevention, it contributes to overcome challenges identified in the literature review. Decentralizing DOTS in Ethiopia and implementing DOTS in Brazil were key recommendations to overcome problems of access and treatment default.

CONCLUSIONS

DOTS and Community DOTS cannot solve every identified Tuberculosis challenge, but together they complement each other. Both strategies need to be tailored to site's challenges.

An integrative theory-driven framework for evaluating travel training programs

Since the 1970s, travel training programs, which provide a short-term training to people with disabilities and older people to teach them independent travel skills required to use fixed-route transportation, have spread across the United States. But the authors note that currently, there is no integrative framework for evaluating the training programs, although it is crucial for improving program implementation and developing knowledge and theories related to travel training. Therefore, this research aims to build an integrative theory-driven evaluation framework of the programs on the basis of prior studies on travel training and the literature on program evaluation and learning and training theories. The framework considers (1) a wide range of key elements related to the delivery systems and outcomes of travel training; (2) diverse stakeholders that engage in designing, operating, and assessing travel training; and (3) the short-term, intermediate, and long-term outcomes of the programs. Based on the framework, the authors develop a flexible logic model for travel training programs to help scholars and practitioners design and conduct actual evaluation studies. Thus, this research is expected to make theoretical and practical contributions to theory-driven program evaluation and travel training programs.

Using a Theory of Change to Guide Grant Monitoring and Grantmaking

Charitable foundations play a significant role in advancing public health, funding billions of dollars in health grants each year. Evaluation is an important accountability tool for foundations and helps ensure that philanthropic investments contribute to the broader public health evidence base. While commitment to evaluation has increased among foundations over the past few decades, effective use of evaluation findings remains challenging. To facilitate use of evaluation findings among philanthropic organizations, evaluators can incorporate the foundation's theory of change-an illustration of the presumed causal pathways between a program's activities and its intended outcomes-into user-friendly products that summarize evaluation findings and recommendations. Using examples from the evaluation of the Kansas Health Foundation's Healthy Living Focus Area, we present a mapping technique that can be applied to assess and graphically depict alignment between program theory and program reality, refine the theory of change, and inform grantmaking.

Investing in a healthy lifestyle strategy: is it worth it?

OBJECTIVES

In Quebec, various actors fund activities aimed at increasing physical activity, improving eating habits and reducing smoking. The objective was to evaluate how effective does the healthy lifestyle habits promotion (HLHP) strategy need to be to make to offset its costs.

METHODS

First, we built the logic model of the HLHP strategy. We then assessed the strategy's total cost as well as the direct health care expenditures associated with lifestyle-related risk factors (smoking, physical inactivity, insufficient intake of fruits and vegetables, obesity and overweight). Finally, we estimated the break-even point beyond which the economic benefits of the HLHP strategy would outweigh its costs.

RESULTS

The HLHP strategy cost for 2010-2011 was estimated at $110 million. Direct healthcare expenditures associated with lifestyle-related risk factors were estimated at $4.161 billion. We estimated that 47 % of these expenditures were attributable to these risk factors.

CONCLUSIONS

We concluded that the HLHP strategy cost corresponded to 5.6 % of the annual healthcare expenditures attributable to these risk factors. This study compared the economic value of HLHP activities against healthcare expenditures associated with targeted risk factors.

Nurse practitioners, canaries in the mine of primary care reform

A strong and effective primary care capacity has been demonstrated to be crucial for controlling costs, improving outcomes, and ultimately enhancing the performance and sustainability of healthcare systems. However, current challenges are such that the future of primary care is unlikely to be an extension of the current dominant model. Profound environmental challenges are accumulating and are likely to drive significant transformation in the field. In this article we build upon the concept of "disruptive innovations" to analyze data from two separate research projects conducted in Quebec (Canada). Results from both projects suggest that introducing nurse practitioners into primary care teams has the potential to disrupt the status quo. We propose three scenarios for the future of primary care and for nurse practitioners' potential contribution to reforming primary care delivery models. In conclusion, we suggest that, like the canary in the coal mine, nurse practitioners' place in primary care will be an indicator of the extent to which healthcare system reforms have actually occurred.

A theoretical evaluation of a youth mental health court program model

Mental health courts are a promising new approach to addressing the overrepresentation of mental health needs among offender populations, yet little is known about how they facilitate change, particularly for youth. The current study reports on a process evaluation of a youth mental health court in Toronto, Canada. Drawing upon observations of the court and interviews with key informants, we developed a program model of the court and explored its implementation within the context of empirical evidence for treating justice-involved youth. Findings revealed that the proposed mechanism of change, which focuses on reducing recidivism through the treatment of mental health needs, should also consider factors directly related to offending behavior. Findings further highlight several strengths of the program, including the program's supportive environment and ability to engage and link youth and families with treatment. Areas for continued growth include the need for comprehensive protections of legal rights.

Integrated Primary Care Teams (IPCT) pilot project in Quebec: a protocol paper

INTRODUCTION

The overall aim of this project is to help develop knowledge about primary care delivery models likely to improve the accessibility, quality and efficiency of care. Operationally, this objective will be achieved through supporting and evaluating 8 primary care team pilot sites that rely on an expanded nursing role within a more intensive team-based, interdisciplinary setting.

METHODS AND ANALYSIS

The first research component is aimed at supporting the development and implementation of the pilot projects, and is divided into 2 parts. The first part is a logical analysis based on interpreting available scientific data to understand the causal processes by which the objectives of the intervention being studied may be achieved. The second part is a developmental evaluation to support teams in the field in a participatory manner and thereby learn from experience. Operationally, the developmental evaluation phase mainly involves semistructured interviews. The second component of the project design focuses on evaluating pilot project results and assessing their costs. This component is in turn made up of 2 parts. Part 1 is a pre-and-post survey of patients receiving the intervention care to analyse their care experience. In part 2, each patient enrolled in part 1 (around 4000 patients) will be matched with 2 patients followed within a traditional primary care model, so that a comparative analysis of the accessibility, quality and efficiency of the intervention can be performed. The cohorts formed in this way will be followed longitudinally for 4 years.

ETHICS AND DISSEMINATION

The project, as well as all consent forms and research tools, have been accepted by 2 health sciences research ethics committees. The procedures used will conform to best practices regarding the anonymity of patients.

Developing and Optimising the Use of Logic Models in Systematic Reviews: Exploring Practice and Good Practice in the Use of Programme Theory in Reviews

BACKGROUND

Logic models are becoming an increasingly common feature of systematic reviews, as is the use of programme theory more generally in systematic reviewing. Logic models offer a framework to help reviewers to 'think' conceptually at various points during the review, and can be a useful tool in defining study inclusion and exclusion criteria, guiding the search strategy, identifying relevant outcomes, identifying mediating and moderating factors, and communicating review findings.

METHODS AND FINDINGS

In this paper we critique the use of logic models in systematic reviews and protocols drawn from two databases representing reviews of health interventions and international development interventions. Programme theory featured only in a minority of the reviews and protocols included. Despite drawing from different disciplinary traditions, reviews and protocols from both sources shared several limitations in their use of logic models and theories of change, and these were used almost unanimously to solely depict pictorially the way in which the intervention worked. Logic models and theories of change were consequently rarely used to communicate the findings of the review.

CONCLUSIONS

Logic models have the potential to be an aid integral throughout the systematic reviewing process. The absence of good practice around their use and development may be one reason for the apparent limited utility of logic models in many existing systematic reviews. These concerns are addressed in the second half of this paper, where we offer a set of principles in the use of logic models and an example of how we constructed a logic model for a review of school-based asthma interventions.

Application of theory-based evaluation for the critical analysis of national biofuel policy: A case study in Malaysia

Theory-based evaluation (TBE) is an effectiveness assessment technique that critically analyses the theory underlying an intervention. Whilst its use has been widely reported in the area of social programmes, it is less applied in the field of energy and climate change policy evaluations. This paper reports a recent study that has evaluated the effectiveness of the national biofuel policy (NBP) for the transport sector in Malaysia by adapting a TBE approach. Three evaluation criteria were derived from the official goals of the NBP, those are (i) improve sustainability and environmental friendliness, (ii) reduce fossil fuel dependency, and (iii) enhance stakeholders' welfare. The policy theory underlying the NBP has been reconstructed through critical examination of the policy and regulatory documents followed by a rigorous appraisal of the causal link within the policy theory through the application of scientific knowledge. This study has identified several weaknesses in the policy framework that may engender the policy to be ineffective. Experiences with the use of a TBE approach for policy evaluations are also shared in this report.

A theory-based evaluation of a community-based funding scheme in a disadvantaged suburban city area

Community-driven development (CDD) initiatives frequently involve funding schemes which are aimed at channelling financial investment into local need and fostering community participation and engagement. This exploratory study examined, through a program theory approach, the design and implementation of a small-scale, community-based fund in Ireland. Observations, documentary analysis, interviews and group discussions with 19 participants were utilized to develop a detailed understanding of the program mechanisms, activities and processes, as well as the experiences of key stakeholders engaged with the funding scheme and its implementation. The findings showed that there were positive perceptions of the scheme and its function within the community. Overall, the availability of funding was perceived by key stakeholders as being beneficial. However, there were concerns over the accessibility of the scheme for more marginalized members of the community, as well as dissatisfaction with the openness and transparency surrounding funding eligibility. Lessons for the implementation of small-scale CDD funds are elaborated and the utility of program theory approaches for evaluators and planners working with programs that fund community-based initiatives is outlined.

An analysis of the adaptability of a professional development program in public health: results from the ALPS Study

Background

Professional development is a key component of effective public health infrastructures. To be successful, professional development programs in public health and health promotion must adapt to practitioners’ complex real-world practice settings while preserving the core components of those programs’ models and theoretical bases. An appropriate balance must be struck between implementation fidelity, defined as respecting the core nature of the program that underlies its effects, and adaptability to context to maximize benefit in specific situations. This article presents a professional development pilot program, the Health Promotion Laboratory (HPL), and analyzes how it was adapted to three different settings while preserving its core components. An exploratory analysis was also conducted to identify team and contextual factors that might have been at play in the emergence of implementation profiles in each site.

Methods

This paper describes the program, its core components and adaptive features, along with three implementation experiences in local public health teams in Quebec, Canada. For each setting, documentary sources were analyzed to trace the implementation of activities, including temporal patterns throughout the project for each program component. Information about teams and their contexts/settings was obtained through documentary analysis and semi-structured interviews with HPL participants, colleagues and managers from each organization.

Results

While each team developed a unique pattern of implementing the activities, all the program’s core components were implemented. Differences of implementation were observed in terms of numbers and percentages of activities related to different components of the program as well as in the patterns of activities across time. It is plausible that organizational characteristics influencing, for example, work schedule flexibility or learning culture might have played a role in the HPL implementation process.

Conclusions

This paper shows how a professional development program model can be adapted to different contexts while preserving its core components. Capturing the heterogeneity of the intervention’s exposure, as was done here, will make possible in-depth impact analyses involving, for example, the testing of program–context interactions to identify program outcomes predictors. Such work is essential to advance knowledge on the action mechanisms of professional development programs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-015-0903-3) contains supplementary material, which is available to authorized users.

A process-based framework to guide nurse practitioners integration into primary healthcare teams: results from a logic analysis

BACKGROUND

Integrating Nurse Practitioners into primary care teams is a process that involves significant challenges. To be successful, nurse practitioner integration into primary care teams requires, among other things, a redefinition of professional boundaries, in particular those of medicine and nursing, a coherent model of inter- and intra- professional collaboration, and team-based work processes that make the best use of the subsidiarity principle. There have been numerous studies on nurse practitioner integration, and the literature provides a comprehensive list of barriers to, and facilitators of, integration. However, this literature is much less prolific in discussing the operational level implications of those barriers and facilitators and in offering practical recommendations.

METHODS

In the context of a large-scale research project on the introduction of nurse practitioners in Quebec (Canada) we relied on a logic-analysis approach based, on the one hand on a realist review of the literature and, on the other hand, on qualitative case-studies in 6 primary healthcare teams in rural and urban area of Quebec.

RESULTS

Five core themes that need to be taken into account when integrating nurse practitioners into primary care teams were identified. Those themes are: planning, role definition, practice model, collaboration, and team support. The present paper has two objectives: to present the methods used to develop the themes, and to discuss an integrative model of nurse practitioner integration support centered around these themes.

CONCLUSION

It concludes with a discussion of how this framework contributes to existing knowledge and some ideas for future avenues of study.

Developing a dual diagnosis service in Cork, Ireland by way of participatory action research (PAR)

Purpose

– Developing a dual diagnosis service in Cork, Ireland by way of participatory action research (PAR) background: internationally there is a growing consensus regarding the ideal of integrated treatment. In Ireland, recommendations identified the need for multi-disciplinary team integration and client participation being central to service development. Such recommendations collectively fit most appropriately with PAR, the methodological and theoretical framework best suited to achieve the objectives of the inquiry. PAR's inclusive philosophy creates processes of negotiation, self-reflexivity and exploration of power issues with the lived experience of communities. Key elements of this approach facilitate the development of emancipatory and participatory democracy whilst highlighting identified social issues through research, learning and action. The paper aims to discuss these issues.

Design/methodology/approach

– Cyclical processes of planning, action, observation and reflection in cycle one have facilitated the introduction of PAR's methodological framework into the existing public health (Health Service Executive) system of primary care addiction and mental health services. Developing stakeholder relationships in decision making processes has been pivotal in cycle 1 as the process of collective engagement evolves. Stakeholders begin to experience their collective participation in the methods adopted and a collective sense of ownership and commitment to the iterative process begins to take shape. Stakeholders in cycle 1 have participated in multiple data generation methods including: informal interviews, planned discussion and focus groups, multidisciplinary team meetings, testimonials, observations and reflections.

Findings

– Cycle 1 of this PAR inquiry has engaged stakeholders (service users and their families, practitioners including; academic/practitioners, a consultant psychiatrist, psychologist, mental health nurses, an occupational therapist, psychotherapists, an acupuncturist, an addiction counsellor, an art therapist) in an integrated process of inquiry. PAR methods adopted in this cycle have facilitated particular dual diagnosis service developments and emerging initiatives (previously unidentified). Actions collaboratively planned for and illustrated in this paper include: the implementation of a psychotherapy group and implementing direct access to an acupuncture clinic.

Originality/value

– Stakeholders collaboratively experience PAR's methodological and theoretical approach which has facilitated service developments in cycle 1 of the inquiry. This sets the stage for the completion of actions already in motion and for further initiatives to continue to evolve as cycle 2 processes emerge.

Program theory-driven evaluation science in a youth development context

Program theory-driven evaluation science (PTDES) provides a useful framework for uncovering the mechanisms responsible for positive change resulting from participation in youth development (YD) programs. Yet it is difficult to find examples of PTDES that capture the complexity of such experiences. This article offers a much-needed example of PTDES applied to Project K, a youth development program with adventure, service-learning and mentoring components. Findings from eight program staff focus groups, 351 youth participants' comments, four key program documents, and results from six previous Project K research projects were integrated to produce a theory of change for the program. A direct logic analysis was then conducted to assess the plausibility of the proposed theory against relevant research literature. This demonstrated that Project K incorporates many of the best practice principles discussed in the literature that covers the three components of the program. The contributions of this theory-building process to organizational learning and development are discussed.