Tailoring a specific medical leadership development program for faculty members: the Lyon-Ottawa experience

The development of leadership skills has been the topic of several position statements over recent decades, and the need of medical leaders for a specific training was emphasized during the COVID-19 crisis, to enable them to adequately collaborate with governments, populations, civic society, organizations, and universities. However, differences persist as to the way such skills are taught, at which step of training, and to whom. From these observations and building on previous experience at the University of Ottawa, a team of medical professors from Lyon (France), Ottawa, and Montreal (Canada) universities decided to develop a specific medical leadership training program dedicated to faculty members taking on leadership responsibilities. This pilot training program was based on a holistic vision of a transformation model for leadership development, the underlying principle of which is that leaders are trained by leaders. All contributors were eminent French and Canadian stakeholders. The model was adapted to French faculty members, following an inner and outer analysis of their specific needs, both contextual and related to their time constraints. This pilot program, which included 10 faculty members from Lyon, was selected to favor interactivity and confidence in older to favor long-term collaborations between them and contribute to institutional changes from the inner; it combined several educational methods mixing interactive plenary sessions and simulation exercises during onescholar year. All the participants completed the program and expressed global satisfaction with it, validating its acceptability by the target. Future work will aim to develop the program, integrate evaluation criteria, and transform it into a graduating training.

Evaluation of Effectiveness of the Unplugged Program on Gambling Behaviours among Adolescents: Study Protocol of the Experimental Controlled Study "GAPUnplugged"

Gambling risk behaviour is an emerging problem among adolescents. "Unplugged" is an effective Social Influence curriculum for preventing substance use among students. This study aims to develop and test a new component focused on gambling added to the Unplugged program. Schools of Piedmont region and Rome city were invited to participate in the study. A self-completed anonymous questionnaire including questions on socio-demographic characteristics, addictive behaviours, beliefs, attitudes and risk perceptions about gambling, normative perceptions, parental practices, school climate, refusal skills, impulsiveness, self-esteem, antisocial behaviours and sensation seeking was prepared for baseline and follow-up surveys. The protocol of the study was submitted and approved by the Novara Ethical Committee and registered in ClinicalTrials.gov (NCT05630157, Protocol ID: 080.742, 11/17/2022). Twenty-nine schools accepted to participate in the study. Sixty-three classes (1325 students) satisfied the eligibility criteria for intervention and were allocated to the intervention arm, and the other 61 (1269 students) were allocated to the control arm. Because of drop-out, absentees, refusals, and invalid questionnaires, data on 1874 students (998 in the intervention and 876 in the control arm), were available for the analysis at baseline. Data management of follow-up questionnaires is in progress. Results of the present study will be useful to clarify the effectiveness of prevention interventions in reducing gambling behaviours among adolescents. Moreover, this will be the first experience of evaluating a new component focused on a different risk behaviour, added to a curriculum previously shown as effective on other risk behaviours.

Outcome evaluation of the p drivers Program: Randomised controlled trial of a program to improve safe driving among novice drivers

BACKGROUND

Globally, road traffic crashes are the leading cause of death for young adults. The P Drivers Project was a trial of a behavioural change program developed for, and targeted at, young Australian drivers in their initial months of solo driving when crash risk is at its highest.

METHODS

In a parallel group randomised controlled trial, drivers (N = 35,109) were recruited within 100 days of obtaining their probationary licence (allowing them to drive unaccompanied) and randomised to an intervention or control group. The intervention was a 3 to 6-week multi-stage driving behaviour change program (P Drivers Program). Surveys were administered at three time points (pre-Program, approximately one month post-Program and at 12 months after). The outcome evaluation employed an on-treatment analysis comprising the 2,419 intervention and 2,810 control participants who completed all required activities, comparing self-reported crashes and police-reported casualty crashes (primary outcome), infringements, self-reported attitudes and behaviours (secondary outcomes) between groups.

RESULTS

The P Drivers Program improved awareness of crash risk factors and intentions to drive more safely, relative to the controls; effects were maintained after 12-months. However, the Program did not reduce self-reported crashes or police-reported casualty crashes. In addition, self-reported violations, errors and risky driving behaviours increased in the intervention group compared to the control group as did recorded traffic infringements. This suggests that despite the Program increasing awareness of risky behaviour in novice drivers, behaviour did not improve. This reinforces the need to collect objective measures to accompany self-reported behaviour and intentions.

CONCLUSIONS

The P Drivers Program was successful in improving attitudes toward driving safety but the negative impact on behaviour, lack of effect on crashes, and the large loss to follow-up fail to support the use of a post-licensing behaviour change program to improve novice driver behaviour and reduce crashes.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry: 363,293 (ANZCTR, 2012).

Program Evaluation's Path to Greater Policy Relevance: Learning From Rossi's Iron Laws

In a 1987 article, Peter R. Rossi promulgated "The Iron Law of Evaluation and Other Metallic Rules." The Metallic Laws were meant as an informal (and humorous) overstatement of the weakness of contemporary evaluations of social programs. Rossi' s underlying worry was not so much about the state of evaluation technology in the abstract, but, rather, in its inability to advance our broad understanding of social problems and what to do about them---in other words, to make evaluation policy relevant. Rossi attributed the continuing failure to develop successful "large-scale social programs" to the failure to build a strong knowledge base for this kind of "social engineering." The qualities of studies that enable such accumulated learning are variously labeled "external validity," "generalizability," "applicability," or "transferability." This Special Issue includes five papers that seek to explore and apply this understanding.

Facilitated engagement approach: A novel approach to guide mentor conversations

BACKGROUND

Mentorship has been recognized as a strategy to develop leadership competencies in clinical leaders and has been integrated into leadership programs. However, there are few published frameworks to guide mentor conversations with mentees training to assume nursing leadership roles.

OBJECTIVE

This study explores mentors' perceptions of 6-month mentorship, a component of the Strengths-Based Nursing Leadership program, the effectiveness of the Facilitated Engagement Approach, a pedagogical strategy developed to facilitate conversation between the mentor and mentee, and the impact of mentorship on leadership practice of mentees.

METHODS

A mixed method qualitative and quantitative approach was used with semi-structured interviews and bi-weekly survey among mentors. Data were thematically analyzed.

RESULTS

Mentors described the use of the Facilitated Engagement Approach to guide their mentorship conversations and found it to be effective in mentoring program participants. Mentors described techniques used and the process of deepening a reflective mentorship conversation. The Spiraling Process (58 %) and the Story Sharing Process (74 %) were integral aspects of the Facilitated Engagement Approach used. Mentorship was found to be impactful in that mentors reported a change in their mentee.

CONCLUSIONS

The Facilitated Engagement Approach was an effective tool for mentees and mentors to develop leadership capacities.

A comprehensive program in academic health professions to enhance faculty recruitment, mentoring, productivity, and retention: The case of RCSI's StAR program

Top faculty talent recruitment, mentoring, productivity, and retention are paramount for organizational success among institutions of higher learning. Programs would do well to treat these various aspects of faculty management/development as inextricably linked to one another, rather than viewing recruitment or retention in a vacuum. The Strategic Academic Recruitment (StAR) program at the Royal College of Surgeons in Ireland (RCSI) University of Medicine and Health Sciences in Dublin was founded to bear these things, along with best practices in faculty development, in mind to enhance organizational effectiveness. This paper provides some background, description, and outcomes of the program thus far, revealing positive trends in scholarly productivity, teaching, program faculty commitment, and the development of future leaders for the institution, even while further evaluation and continued quality improvement for the StAR initiative are called for. It is hoped that the details provided here can be helpful for other academic organizations as they consider any of various initiatives aimed to attract high-quality labor capital, position those faculty for success, and enhance organizational effectiveness and reputation.

Community Health Workers: Improving Home Visiting Engagement of High-Risk Birthing People in Segregated Neighborhoods

CONTEXT

Racial and ethnic disparities in perinatal health remain a public health crisis. Despite improved outcomes from home visiting (HV) participation during pregnancy, most eligible individuals of color do not engage. Neighborhood segregation, a manifestation of structural racism, may impose constraints on engaging eligible individuals in HV.

OBJECTIVE

To examine whether race, ethnicity, and/or language-concordant community health workers (CHWs) increased HV engagement for birthing people in segregated neighborhoods.

DESIGN

Program evaluation using administrative linked data from birth records, Medicaid claims, and HV program participation. Strong Beginnings (SB), a program with HV provided by CHWs working with nurses and social workers, was compared with the Maternal Infant Health Program (MIHP), a state Medicaid-sponsored HV program without CHW involvement. Data were analyzed using χ 2 tests and Poisson regressions.

PARTICIPANTS

A total of 4560 individuals with a Medicaid-eligible birth between 2016 and 2019, including 1172 from SB and 3388 from the MIHP.

MAIN OUTCOME MEASURES

Penetration (percentage of participants in HV among all Medicaid-eligible individuals across quintiles of neighborhood segregation) and dosage (the total number of home visits from both CHWs and nurses/social workers, and then restricted to those from nurses/social workers).

RESULTS

SB penetrated more segregated neighborhoods than the MIHP (58.4% vs 48.3%; P < .001). SB participants received a higher dosage of home visits (mean [SD]: 11.9 [6.1]) than MIHP participants (mean [SD]: 4.4 [2.8], P < .001). Importantly, CHWs did not replace but moderately increased home visits from nurses and social workers (51.1% vs 35.2% with ≥5 intervention visits, P < .001), especially in more segregated neighborhoods.

POLICY IMPLICATION

Community-informed HV models intentionally designed for people facing disparities may help facilitate program outreach to segregated neighborhoods with concentrated deprivation and reduce racial and ethnic disparities.

CONCLUSIONS

An HV program provided by CHWs working with nurses and social workers was associated with an increase in penetration and dosage in segregated neighborhoods, compared with HV without CHW involvement. This underscores the value of CHWs partnering with licensed professional workers in improving HV engagement in disadvantaged communities.

Randomized Control Trial of a School-Based Curriculum that Teaches About Multiple Forms of Abuse

Most school-based prevention curricula for young children fail to address multiple types of abuse and limit instruction to a single day, despite evidence that polyvictimization is common and children learn better when allowed to practice material repeatedly. This study utilized a cluster randomized control trial design to evaluate a multi-abuse prevention program, the Monique Burr Foundation for Children (MBF) Child Safety Matters®, based on varying lesson structure. Participants included nine Florida schools consisting of 843 children in grades K-2. Schools were randomized within cluster to implement in two lessons, four lessons, or after study data collection (i.e., control group). Lessons averaged 34 minutes (SD = 8.8 minutes) in length for 2-lesson group and 23.6 minutes (SD = 6.9 minutes) for the 4-lesson condition. Knowledge was assessed before implementation and on average 11 weeks after implementation. There were no statistical differences between clusters. Controlling for pre-test scores, schools in the four-lesson group scored highest on a measure of knowledge of potentially risky and unsafe situations (M = 69.68, SE = .80) on post-test, followed by schools in the two-lesson group (M = 67.31, SE = .77), followed by schools in the control group (M = 62.92, SE = .76). Results support use of more frequent, shorter lessons for prevention programs and the promise of addressing multiple forms of child victimization.

Feasibility of a quality-improvement program based on routinely collected health outcomes in Dutch primary care physical therapist practice: a mixed-methods study

BACKGROUND

This study evaluates the feasibility of a nine-month advanced quality-improvement program aimed at enhancing the quality of care provided by primary care physical therapists in the Netherlands. The evaluation is based on routinely collected health outcomes of patients with nonspecific low back pain, assessing three feasibility domains: (1) appropriateness, feasibility, and acceptability for quality-improvement purposes; (2) impact on clinical performance; and (3) impact on learning and behavioral change.

METHODS

A mixed-methods quality-improvement study using a concurrent triangulation design was conducted in primary care physical therapist practice. Feedback reports on the processes and outcomes of care, peer assessment, and self-assessment were used in a Plan-Do-Study-Act cycle based on self-selected goals. The program's appropriateness, feasibility, and acceptability, as well as the impact on clinical performance, were evaluated using the Intervention Appropriate Measure, Feasibility Intervention Measure, Acceptability Intervention Measure (for these three measure, possible scores range from 4 to 20), and with a self-assessment of clinical performance (scored 0-10), respectively. The impact on learning and behavioral change was evaluated qualitatively with a directed content analysis.

RESULTS

Ten physical therapists from two practices participated in this study. They rated the program with a mean of 16.5 (SD 1.9) for appropriateness, 17.1 (SD 2.2) for feasibility, and 16.4 (SD 1.5) for acceptability. Participants gave their development in clinical performance a mean score of 6.7 (SD 1.8). Participants became aware of the potential value of using outcome data and gained insight into their own routines and motivations. They changed their data collection routines, implemented data in their routine practice, and explored the impact on their clinical behavior.

CONCLUSIONS

This explorative study demonstrated that a quality-improvement program, using health outcomes from a national registry, is judged to be feasible.

IMPACT STATEMENT

This study provides preliminary evidence on how physical therapists may use health outcomes to improve their quality, which can be further used in initiatives to improve outcome-based care in primary physical therapy.

A mixed methods evaluation of the impact of ECHO® telementoring model for capacity building of community health workers in India

INTRODUCTION

India has the largest cohort of community health workers with one million Accredited Social Health Activists (ASHAs). ASHAs play vital role in providing health education and promoting accessible health care services in the community. Despite their potential to improve the health status of people, they remain largely underutilized because of their limited knowledge and skills. Considering this gap, Extension for Community Healthcare Outcomes (ECHO)® India, in collaboration with the National Health System Resource Centre (NHSRC), implemented a 15-h (over 6 months) refresher training for ASHAs using a telementoring interface. The present study intends to assess the impact of the training program for improving the knowledge and skills of ASHA workers.

METHODS

We conducted a pre-post quasi-experimental study using a convergent parallel mixed-method approach. The quantitative survey (n = 490) assessed learning competence, performance, and satisfaction of the ASHAs. In addition to the above, in-depth interviews with ASHAs (n = 12) and key informant interviews with other stakeholders (n = 9) examined the experience and practical applications of the training. Inferences from the quantitative and qualitative approaches were integrated during the reporting stage and presented using an adapted Moore's Expanded Outcomes Framework.

RESULTS

There was a statistically significant improvement in learning (p = 0.038) and competence (p = 0.01) after attending the training. Participants were satisfied with the opportunity provided by the teleECHO™ sessions to upgrade their knowledge. However, internet connectivity, duration and number of participants in the sessions were identified as areas that needed improvement for future training programs. An improvement in confidence to communicate more effectively with the community was reported. Positive changes in the attitudes of ASHAs towards patient and community members were also reported after attending the training. The peer-to-peer learning through case-based discussion approach helped ensure that the training was relevant to the needs and work of the ASHAs.

CONCLUSIONS

The ECHO Model ™ was found effective in improving and updating the knowledge and skills of ASHAs across different geographies in India. Efforts directed towards knowledge upgradation of ASHAs are crucial for strengthening the health system at the community level. The findings of this study can be used to guide future training programs. Trial registration The study has been registered at the Clinical Trials Registry, India (CTRI/2021/10/037189) dated 08/10/2021.

A proposal to evaluate the management of tuberculosis programs: a qualitative, evaluability assessment in the border region of Brazil and Venezuela

This study aims to analyze the feasibility of building an evaluative model for the management of the Tuberculosis Prevention and Control Program in the State of Roraima, located on the border between Brazil and Venezuela. This is an evaluability assessment, a type of study used as a pre-evaluation of the development and implementation stages of a program, as well as throughout its execution. The study was developed in stages comprising the: (i) definition of the intervention to be analyzed and its objectives and goals; (ii) construction of the intervention logical model; (iii) screening of parties interested in the evaluation; (iv) definition of the evaluative questions; and (v) design of the evaluation matrix. Four priority components were defined for the evaluation: management of the organization and implementation of tuberculosis (TB) prevention and control policy; epidemiological surveillance management; care network management; and management of expected/achieved results. In this model, and based on theoretical references, we defined the necessary resources, activities, outputs, outcomes, and the expected impact for each of the policy management components. The management of the TB control program is feasible for evaluation based on the design of its components, the definition of structure and process indicators, and relevant results for the analysis of the management of TB prevention and control actions, as well as its influence on compliance with the agreed indicators and targets aiming at eradicating the disease by 2035.

Statistical estimation of fatal and serious injuries saved by iRAP protocols in 74 countries

OBJECTIVE

Road traffic crashes cause 1.19 million deaths and millions more injuries annually. The persistently high burden has drawn attention from national and international stakeholders worldwide. Unsafe road infrastructure is one of the major risk factors for traffic safety, particularly in low- and middle-income countries.

METHODS

Aiming to eliminate high-risk roads in all countries, the International Road Assessment Programme (iRAP) developed a robust and evidence-based approach to support country transportation agencies.

RESULTS

Thus far, the iRAP protocols have been used to collect 1.8 million kilometers of Crash Risk Mapping and 1.5 million kilometers of Star Rating and FSI estimations in 128 countries. Deploying an observational before-and-after (or pre-post) study design, this report estimated the fatal and series injuries (FSI) saved through use of the iRAP protocols. The study is based on 441,753 kilometers of assessed roads from 1,039 projects in 74 countries. Our results show that the implementation of iRAP's proposed countermeasures saves about 159,936 FSI annually. Throughout the lifetime of the implemented countermeasures, a total of 3.2 million FSI could be saved.

CONCLUSION

While quantifying the success of the iRAP protocols, our results suggest an opportunity to save many millions more lives on the roads through expanding iRAP implementation to more regions and countries.

Program evaluation of a school-based mental health and wellness curriculum featuring yoga and mindfulness

BACKGROUND

Interest in the effectiveness of mindfulness-based interventions such as yoga in primary schools has grown. Evidence shows promise, as youth who engage in yoga to promote mindfulness show improved coping skills, increased socio-emotional competence and prosocial skills, academic performance, attention span, and ability to deal with stress.

OBJECTIVE

This study reports the results of a program evaluation of a universal health and wellness curriculum, Pure Power, designed to teach youth yoga techniques, mindfulness, and emotion regulation.

METHODS

A non-randomized comparison design examined outcomes among participants from schools that completed the intervention with highest fidelity of implementation (n = 461) and from students in matched comparison schools (n = 420). Standard measures of coping, emotion regulation and emotion dysregulation, spelling, and math achievement were collected.

RESULTS

Analyses suggest the youth in the intervention schools demonstrated relative improvement on measures of emotion regulation, spelling, and math.

CONCLUSIONS

Challenges in implementation in real-life settings are vital to identify. The data provide some real-world evidence for the effectiveness of a universal health and wellness curriculum on emotion regulation and positive academic outcomes. Training school staff to deliver the intervention may foster implementation. Future research should test the effectiveness of who delivers the intervention; for example, teacher-delivered groups vs. other wellness personnel.

Implementing a mental health app library in primary care: A feasibility study

Confronted with a wide range of digital health tools (DHT), professionals and patients need guidance to use these tools correctly and optimize health management. In the fall of 2020, a DHT library developed by Quebec-based company TherAppX was implemented in 22 institutions. The library was designed to enable healthcare professionals to use DHT in clinical care. The purpose of the current study was to assess the feasibility of implementing the library, including user experience, changes in DHT recommendation habits, and factors that helped or hindered the implementation process. A multi-methods design focusing on secondary use of quantitative data collected by TherAppX and semi-structured interviews with users was employed. While the quantitative analyses indicated infrequent use of the library, qualitative analyses highlighted several factors that hindered its implementation, including certain library and user characteristics and the unprecedented context of the COVID-19 pandemic. Nevertheless, the quantitative analyses confirmed interest in DHT and their usefulness during follow-ups. The results revealed a marginally significant pre-post changes in the frequency with which DHT were recommended. This study helped identify areas for improvements and indicates that further evaluation is needed. Future implementations would benefit from ensuring optimal conditions for a successful implementation.

Learner Assessment and Program Evaluation: Supporting Precision Education

ABSTRACT

Precision education (PE) systematically leverages data and advanced analytics to inform educational interventions that, in turn, promote meaningful learner outcomes. PE does this by incorporating analytic results back into the education continuum through continuous feedback cycles. These data-informed sequences of planning, learning, assessing, and adjusting foster competence and adaptive expertise. PE cycles occur at individual (micro), program (meso), or system (macro) levels. This article focuses on program- and system-level PE.Data for PE come from a multitude of sources, including learner assessment and program evaluation. The authors describe the link between these data and the vital role evaluation plays in providing evidence of educational effectiveness. By including prior program evaluation research supporting this claim, the authors illustrate the link between training programs and patient outcomes. They also describe existing national reports providing feedback to programs and institutions, as well as 2 emerging, multiorganization program- and system-level PE efforts. The challenges encountered by those implementing PE and the continuing need to advance this work illuminate the necessity for increased cross-disciplinary collaborations and a national cross-organizational data-sharing effort.Finally, the authors propose practical approaches for funding a national initiative in PE as well as potential models for advancing the field of PE. Lessons learned from successes by others illustrate the promise of these recommendations.

Feasibility of Intervention Program to Prevent Adolescent Health-Compromising Behaviors

Most people start experimenting with and/or initiating health-compromising behaviors and establishing behavioral patterns during adolescence. Possible selves and episodic future thinking have been used to foster behavioral changes. In this study, we aimed to (1) develop a program incorporating possible selves and episodic future thinking to decrease the risks associated with health-compromising behaviors, and (2) examine the feasibility and fidelity of implementing this intervention among Taiwanese adolescents. A novel combination of existing intervention approaches tailored for the Taiwanese context was applied to develop a school-based intervention, which we named the Future Mapping Master program. This program contains four core units: exploring the self, setting goals, developing strategies and problem solving, and understanding the threats of health-compromising behaviors. It includes daily episodic future thinking visualization exercises. The feasibility evaluation was conducted with 88 adolescents aged 12-14 years. Feedback from teachers and students supported the proposed program's feasibility and revealed favorable reactions to the intervention. The Future Mapping Master program is feasible and offers promising strategies for implementation in Taiwanese school settings.

Engaging stakeholders to retrospectively discern implementation strategies to support program evaluation: Proposed method and case study

BACKGROUND

Availability of evidence-based practices (EBPs) is critical for improving health care outcomes, but diffusion can be challenging. Implementation activities increase the adoption of EBPs and support sustainability. However, when implementation activities are a part of quality improvement processes, evaluation of the time and cost associated with these activities is challenged by the need for a correct classification of these activities to a known taxonomy of implementation strategies by implementation actors.

DESIGN

Observational study of a four-stage, stakeholder-engaged process for identifying implementation activities and estimating the associated costs.

RESULTS

A national initiative in the Veterans Health Administration (VHA) to improve Advance Care Planning (ACP) via Group Visits (ACP-GV) for rural veterans identified 49 potential implementation activities. Evaluators translated and reduced these to 14 strategies used across three groups with the aid of implementation actors. Data were collected to determine the total implementation effort and applied cost estimates to estimate the budget impact of implementation for VHA.

LIMITATIONS

Recall bias may influence the identification of potential implementation activities.

CONCLUSIONS

This process improved understanding of the implementation effort and allowed estimation of ACP-GV 's budget impact.

IMPLICATIONS

A four-stage, stakeholder-engaged methodology can be applied to other initiatives when a pragmatic evaluation of implementation efforts is needed.

Protocol for an interview-based method for mapping mental models using causal-loop diagramming and realist interviewing

Causal-loop diagramming, a method from system dynamics, is increasingly used in evaluation to describe individuals' understanding of how policies or programs do or could work ("mental models"). The use of qualitative interviews to inform model development is common, but guidance for how to design and conduct these interviews to elicit causal information in participant mental models is scant. A key strength of semi-structured qualitative interviews is that they let participants speak freely; they are not, however, designed to elicit causal information. Moreover, much of human communication about mental models-particularly larger causal structures such as feedback loops-is implicit. In qualitative research, part of the skill and art of effective interviewing and analysis involves listening for information that is expressed implicitly. Similarly, a skilled facilitator can recognize and inquire about implied causal structures, as is commonly done in group model building. To standardize and make accessible these approaches, we have formalized a protocol for designing and conducting semi-structured interviews tailored to eliciting mental models using causal-loop diagramming. We build on qualitative research methods, system dynamics, and realist interviewing. This novel, integrative method is designed to increase transparency and rigor in the use of interviews for system dynamics and has a variety of potential applications.

Designing a questionnaire with retrospective pre-post items: Format matters

Evaluators are frequently asked to evaluate educational interventions that are one-time events. The Retrospective Pretest (RPT) methodology is well suited for these circumstances and the evaluation literature is replete with discussions about the pros and cons of this approach. In RPT, program participants rate their attitudes or knowledge now, and also rate how they were before participation in the intervention. The difference between now and before ratings constitutes the measure of change. Little published literature exists about whether the layout of RPT items within the evaluation questionnaire yields different results. This study compared six different layouts using a sample of 1941 caregivers who participated in one of 96 training workshops. The layouts were compared on inattentiveness, unexpected decline in perceived knowledge, and the degree of before-now change. Findings show that design matters. The best performing layout is where items are placed in the center with before response options on the left of the page and the now response options on the right. Results point to the need for evaluators to pay attention to RPT layout, and for the field to establish criteria for assessing survey layout quality. The unexpectedly high rate of inattentiveness calls for evaluators to pay more attention identifying and addressing this in their survey data.

Outcomes for Adults With Intellectual and Developmental Disabilities Receiving Long-Term Services and Supports: A Systematic Review of the Literature

The impact of long-term services and supports on the quality of life of adults with intellectual and developmental disabilities (IDD) is not well understood given the highly complex nature of researching this topic. To support future research addressing this topic, we conducted a systematic literature review of studies addressing outcomes of adults with IDD receiving long-term services and supports. Results of this review describe current outcomes for adults with IDD who receive long-term services and supports and can be used to inform program evaluation, policy development, and future research.

An evaluation of an employment assistance program focused on people living with HIV in Toronto, Canada

Unemployment is more common among people living with HIV (PLWH) compared to the general population. PLWH who are employed have better physical and mental health outcomes compared to unemployed PLWH. The main objective of this mixed-methods study was to conduct a program evaluation of Employment Action (EACT), a community-based program that assists PLWH in Toronto, Ontario, Canada to maintain meaningful employment. We extracted quantitative data from two HIV services databases used by EACT, and collected qualitative data from 12 individuals who had been placed into paid employment through EACT. From 131 clients included in the analysis, 38.1% (n = 50) maintained their job for at least 6 weeks within the first year of enrollment in the EACT program. Gender, ethnicity, age, and first language did not predict employment maintenance. Our interviews highlighted the barriers and facilitators to effective service delivery. Key recommendations include implementing skills training, embedding PLWH as EACT staff, and following up with clients once they gain employment. Investment in social programs such as EACT are essential for strengthening their data collection capacity, active outreach to service users, and sufficient planning for the evaluation phase prior to program implementation.

If you really love nothing. Evaluating second-order factors in the case of Italian constitutional referendum of 2020

The paper evaluates several second-order factors to explain the outcomes of the 2020 constitutional referendum held in Italy. This particular referendum serves as a relevant case study to assess the significance of second-order factors, given its unique characteristics, including the simplicity of the referendum question regarding the reduction in the number of parliamentarians (MPs), the minimal impact on public finances resulting from this reduction, and the subsequent decrease in territorial democratic representation. While it might have been expected that a significant majority would vote in favour of the "No" option, thus preserving the current levels of territorial coverage (and democratic representation) of MPs, the actual results saw a substantial majority in favour of the "Yes" vote (69%). Our argument posits that the overwhelming prevalence of the "Yes" vote (to reduce the number of MPs), especially in poorer areas of the country, can be attributed to specific factors that influence individual evaluations in a direct-democratic setting. In greater detail, by using cross-sectional data, the paper tests the role of socio-economic condition, trust in institutions and political participation in affecting the referendum outcome. The results of our empirical analysis confirm our hypotheses, demonstrating that second-order factors indeed influenced the referendum's outcome. Specifically, our analysis reveals that: (i) a higher socio-economic condition could generate a higher share of "No" votes; (ii) a higher trust in institutions could lead to an increase in the share of "No" votes; finally, that (iii) an increase in political participation could produce a decrease in the share of "No" votes. In the concluding section of the paper, we discuss how this analysis contributes new insights to the study of voting behaviour in direct-democratic contexts.

Sustaining health obesity prevention programs: Lessons from real-world population settings

Current understanding of what sustains public health programs is derived primarily from evaluations of programs delivered for only a few years. To improve program planning and knowledge on program sustainment, we examined real-world population programs that have been delivered for >2 years. Our focus was physical activity and nutrition programs for obesity prevention. We identified programs through published literature and searched through publicly available information on the theoretical factors determining sustainment. We reviewed 90 programs and explored intervention characteristics, provider, delivery and process level factors, and the outer environment influences. Programs were sustained on average 15 years and more frequently characterised as behavioural change skills programs targeting children, funded by non-government organisations and delivered by community organisations. Most programs had undergone some modification (n = 55). Differences between programs were observed across key factors: Programs sustained 15+ years were mainly behaviour skills programs (n = 21); and characterised by interactive designs (i.e., face-to-face or digital) over static innovations (i.e., print material) compared with those sustained <15 years (p = 0.024). While government funding supported many of the programs (n = 45;), those sustained 15+ years were more likely to have been commercially funded (p = 0.044); and were less likely to have current community involvement (p = 0.013). Differences in ownership and funding were also observed across countries. While multiple factors may influence program sustainment, longer delivered programs where distinguished by their program characteristics and outer contextual factors, suggesting such factors may have an important role in sustaining programs in the longer-term.

Partnership in Cancer Research (PCAR) Program Increases Medical Student Knowledge and Confidence to Perform Cancer Research

Arkansas has a high cancer burden, and a pressing need exists for more medical students to pursue oncology as a career. The Partnership in Cancer Research (PCAR) program provides a summer research experience at the University of Arkansas for Medical Sciences for 12 medical students who have completed their first year of medical training. A majority of participants spend time pursuing cancer research in basic science, clinical, or community-based research. Students report on their research progress in an interactive "Live from the Lab!" series and assemble a final poster presentation describing their findings. Other activities include participation in a moderated, cancer-patient support group online, lecture series on cancer topics, medical simulations, palliative care clinic visit, "Death Over Dinner" event, and an entrepreneurship competition. Students completed surveys over PCAR's first 2 years in operation to evaluate all aspects of the program. Surveys reveal that students enthusiastically embraced the program in its entirety. This was especially true of the medical simulations which received the highest evaluations. Most significantly, surveys revealed that the program increased cancer knowledge and participant confidence to perform cancer research.

The Chicagoland Free Clinics Consortium: A Model for Student-Run Free Clinic Collaboration

BACKGROUND

While student-run free clinics (SRFCs) play an important role in care for underserved populations, few mechanisms exist to promote collaboration among regional SRFCs.

AIMS

To address this gap, the Chicagoland Free Clinics Consortium (CFCC) was formed to (1) facilitate collaboration between Chicagoland SRFCs, (2) provide innovation grant funding, and (3) host an annual conference.

SETTING AND PARTICIPANTS

In 2018, students from the Pritzker School of Medicine founded the CFCC and partnered with peers from area schools to implement programming.

PROGRAM DESCRIPTION

Between 2018 and 2022, CFCC engaged 23 SRFCs representing all 6 Chicagoland schools, held 4 annual conferences, and distributed $15,423 in grants to 19 projects at 14 SRFC sites.

PROGRAM EVALUATION

A total of 176 students from 5 schools attended the 4 conferences. In 2022, 82 unique participants were surveyed, and 66% (54/82) responded. Eighty percent (43/54) reported they were "more likely to collaborate with other Chicagoland free clinics." In 2022, all grant sites were surveyed and 84% (16/19) responded. Most (87%,14/16) agreed the grant "allowed them to implement a project that would not have otherwise been accomplished" and 21% (4/19) were inter-institutional collaborations.

DISCUSSION

To our knowledge, CFCC is the first student-led organization to promote sustained collaboration across SRFCs in a metropolitan area.

School-based psychosocial and educational interventions for children and adolescents after the 1999 Marmara earthquakes in Turkey: A review on lessons learned

The massive earthquakes experienced in August and November 1999 affected thousands of people in the Marmara region, the most densely populated and industrialized part of Turkey. The humanitarian and economic cost was so enormous, and these earthquakes have changed the Turkish disaster management system and the Turkish people's approach to disasters. Marmara earthquakes are also considered as a milestone in the provision of psychosocial services for disaster victims. This paper aims to review the psychosocial interventions targeting children, adolescents, and their families after the 1999 earthquakes in Turkey. The progression from initial responses to more organized psychosocial interventions is outlined. Conducting the interventions at schools has ensured that thousands of children, teachers, and parents are reached in the most efficient and effective way possible. The significance of the school context in designing psychosocial interventions is highlighted and implications of the lessons learned for traumatic experiences of children and parents are also explored. It is evaluated that these inferences obtained from the Marmara earthquake in Turkey can be used in disasters around the world.

Exploring the curricular and pedagogical decision criteria for research-based learning design in undergraduate studies

Learning design has a multifaceted nature requiring a range of course- and institutional considerations. Analyzing the decision criteria's influence on research-based learning design helps understand the causes of the success/failure of the approach in achieving the teaching goals to improve the study programs. This study explores the interrelationship between the curricular and pedagogical criteria for research-based learning design decisions at the undergraduate level. For this purpose, the DEcision-MAking Trial and Evaluation Laboratory (DEMATEL) method is used to systematically analyze the decisive criteria and their causal relationships. Feedback from education professionals and university professors from Scandinavian universities is used to validate the pedagogy decision framework and provides input into the DEMATEL method. The student's role in the course is identified as the central criterion, featuring the highest level of interactions in the network of curricular and pedagogical decision criteria. Results are supportive of the identified institutional and course-specific criteria as prerequisites for the study outcomes.

Introducing a multicomponent staff training intervention to reduce antipsychotic medication: Care home management pre and post intervention views of systemic impact, and preliminary RE-AIM evaluation

OBJECTIVES

We explored views of care home managers when introducing PROCUIDA-Demencia a mixed-methods two-arm cluster randomised controlled pilot and clinical outcomes study aiming to optimise dementia care by introducing psychosocial interventions to reduce antipsychotic medication in care homes.

METHOD

We undertook secondary analysis of pre and post in-depth interviews conducted in summer 2018 with not-for-profit care home managers in Mexico who were allocated to the intervention group. Transcribed data were thematically analysed. Themes were mapped out with RE-AIM quality appraisal framework (Reach, Effectiveness, Adoption, Implementation and Maintenance) as preliminary evaluation to identify practice and future intervention development and evaluation.

RESULTS

Two pre- and three post-intervention themes were constructed. Participants reported measurable positive impact; one home built a new specialist dementia care unit and others hired a psychologist and psychiatrist to sustain the changes. Antipsychotic medication was reduced for some participating residents which also minimised cost burden on family members.

CONCLUSION

Funding, systemic working across families, clinical and social teams and effective systems of governance are urgently required to sustain models like PROCUIDA-Demencia. The RE-AIM preliminary evaluation outlined care home managers' long-term sustainable practice and positive impact on the dementia care system. These findings might inform staff retention strategies and care home systemic care practices. This evaluation is contributing to the Mexican Alzheimer's and other dementias plan.

Teachers' teaching and professional competences assessment

This research is part of a project led by the Catholic University of Santiago de Guayaquil (Ecuador) and the National Distance Education University - UNED (Spain) aiming at designing a teaching/professional competences training model for teachers. The purpose of this study is to assess the meaning of professional competences and their significance within teachers' teaching and training processes. Using the mixed methods approach, an ad hoc questionnaire was designed and given to discussion groups, achieving participation of more than a hundred teachers. Quantitative data analysis was carried out using the Logit model while Atlas ti 23 allowed analysis of the qualitative data. The results evidenced high scores in the competences evaluated, being leadership and research the lowest. In general, the teachers value, preferably, planning, communication, evaluation, methodology, digital, and tutoring competences, with a lower value among teachers ranging from 25 to 30 years old; those older than 55 stressed the importance of digital and innovation competences. Thus, it is recommended that training programs for teachers should focus on the set of related competences, especially on the digital competence, considering the challenges represented by investigation and pedagogical leadership.

Teaching competence of TVET teachers in the digital age: Implementation and evaluation of a training program in China

TVET teachers need comprehensive competence in response to the societal change of digitalization. Comparing to well-developed preparation programs in general education, the training framework for TVET teacher still need to be examined, especially in school-based TVET system. This study aims to propose a teaching competence framework and investigates its fitness and effectiveness in on-service training. A training program was conducted corresponding to it in a case school in China. The program achieves significant improvement in competence by using pretest and posttest measure, and receives satisfactory feedback from a survey among participating teachers. Factors which may affect training operation and outcome are discussed.

An evaluation of the impact of mitigation policies on health and the economy by managing social distancing during outbreaks

The COVID-19 pandemic has necessitated various unavoidable social restrictions, leading to questions about the effectiveness of public emergency interventions and their impact economic growth. Block et al. (2020) conducted a notably study using an agent-based model to evaluate policies for reducing contact and demonstrated how choices in contact behavior can influence the rate and spread of the virus. However, their approach did not consider the economic consequences of these social restrictions. In response, we propose a set of strategies for governments to plan and evaluate policies during emergencies, aiming to contain infections while minimizing negative economic consequences. Our results indicate that there is no trade-off between containment strategies and economic output loss, making containment measures necessary policy instruments. However, potential trade-offs do emerge when selecting the most effective strategy. In this context, we propose and evaluate various policy alternatives to extreme "social distancing" measures, which can partially restore essential social interactions while preventing economic disasters induced by productivity losses.

Schools within the uMgungundlovu district: Principals' perspectives on categorization and school functionality

School categorization has impaired the functionality of schools and led to dysfunctional teaching and learning processes. This study explores principals' perspectives on categorization and school functionality. The existing literature on school categorization focuses on the geographical areas of schools and resource allocation, ignoring the complexities of learner behavior and other dynamics within schools. Based on a qualitative case study within an interpretive paradigm, this study was conducted in the uMgungundlovu district of KwaZulu-Natal using semi-structured interviews, document reviews, and observations with five purposefully selected principals of five primary schools. Data analysis revealed that schools were paying significant maintenance and bills to sustain themselves without sufficient funding. In addition, the quintile classification indicates that the socioeconomic status of schools weakens the institution's ability to provide intensive development and growth for administrative and learning needs and to the extent that schools generate their resources for the reconstruction of structures and facilities, affecting the well-being of teachers and learners, as well as external factors such as professional development and academic training, which have a significant impact on the delivery of quality education.

Performance management and policy evaluation of information and communication technology graduate program for developing countries

As an educational aid, the Korean government has developed and funded a graduate-level global information telecommunications technology program (ITTP) for ICT officers in developing countries. Questions have arisen about whether the program can achieve its goals and obtain mutual benefits between Korea and participating nations. Using a total quality management (TQM) framework as a unique case analysis, we evaluated ITTP's resources and performance from 2006 to 2014. Our results show that ITTP has focused on the quantitative promotion of students and alums from various countries and their learning of Korea-centric ICT, policies, and culture. In particular, the Korean government's lax evaluation of ITTP performance has led to weak responsibility, ambiguous task identity, missed opportunities for self-improvement, and low participation among students and stakeholders. These findings highlight the need for graduate programs and their stakeholders to reinforce internal and external evaluations of the program's performance and provide feedback for reciprocal development.

Towards transparent decision-making processes within museums: Case study of Museu Nacional d'Art de Catalunya (MNAC)

Understanding that good decision-making is paramount for the success of an organization and recognizing that strategic decision-making inside museums is a topic that has been over-passed by researchers, this paper intends to contribute to the relevance of the subject through the implementation of an innovative tool. Decision-making in museums is normally focused on the intuition, subjectivity, and experience of the curator, who brings all his knowledge to the exhibition programming. However, museums' management and environment are constantly changing, thus the application of this tool would aim to make decisions in a more democratic, transparent, inclusive, and accurate manner. Besides, it will be easier to understand how distinct subjects can work together, demonstrating successful results to improve the decision-making process when programming and measuring temporary exhibition seasons in museums. This paper describes the design and implementation of the Multi-Attribute Utility Theory (MAUT) for the Museu Nacional d'Art de Catalunya (MNAC). It also analyzes and evaluates the decision-making process when scheduling exhibitions for a season.

Implementing trauma-informed care in a special education setting: An initial exploration of a multi-tiered model

Implementing trauma-informed care in a special education environment serving youth from historically marginalized communities with high levels of exposure to potentially traumatic events (PTEs) requires a systematic tiered approach consistent with public health guidelines. Little is known about the implementation of this framework in special education settings where youth have significant emotional and behavioral difficulties. To address this need, a consultant-community partnership was forged between a hospital providing mental health services and a therapeutic day school that serves a special education cooperative. The current case study explores the design and implementation of a three-tiered model of trauma-informed care in a special education setting. This study will address the specific practices implemented at each tier, discuss successes and challenges, and summarize future directions for research, practice, and policy.

A realist evaluation of a multi-component program with disengaged students

Periods spent in the absence of education, employment, or training (NEET) are associated with adverse psychological wellbeing, social marginalisation, and premature mortality. Implementing effective programs to re-engage young people who are classified, or are at risk of becoming NEET, is of importance to these individuals, family, and society. We conducted a realist evaluation to understand how, and under which circumstances a multi-component program may impact the engagement, behavioural, and psychosocial outcomes of disengaged students at risk of becoming NEET. During the early project phase, a narrative review of the literature and key stakeholder discussions were conducted to develop our initial program theories regarding how the program was expected to achieve its outcomes. Participant observations, video footage, and forty-two interviews were then conducted with teachers and students to form context-mechanism-outcome configurations and to refine these theories. Overall, refined program theories relating to positions of authority, the power of collective experience, exploration of possible life directions, constructivist pedagogies and active learning, and the endorsement of an ethic of caring and strengths-based orientation were developed. Collectively, our findings provide a detailed understanding of the architecture of programs that may benefit disengaged students and help inform the design of future programs aimed at reducing disaffection.

Improvements in nutrition knowledge among adults with physical disabilities: A program evaluation of the mindfulness, exercise, and nutrition to optimize resilience (MENTOR) program 2020-2021

BACKGROUND

The Centers for Disease Control and Prevention (CDC) funds a quality improvement (QI) health promotion telewellness program for adults with physical disabilities.

OBJECTIVE

The nutrition component of the parent program, Mindfulness, Exercise, and Nutrition to Optimize Resilience (MENTOR) program, from 2020 to 2021, was evaluated to assess changes in nutrition knowledge of people with physical disabilities.

METHODS

This was a quasi-experimental program evaluation by pre-post survey of participating adults with physical disabilities. Participants attended weekly nutrition classes for eight weeks with optional one-on-one counseling with a Registered Dietitian. Participants completed pre-post General Nutrition Knowledge Questionnaire Revised (GNKQ-R) that were analyzed using paired t-tests. Participants were grouped into quartiles by pre-program GNKQ-R scores for additional comparisons. Program attendance and post-program feedback were also evaluated.

RESULTS

A total of 286 participants completed the MENTOR program, and 31% (n = 89) completed pre-post questionnaires for data analysis. Mean pre-program GNKQ-R scores were 58.4 ± 15.1 and post scores were 64.4 ± 11.3, which demonstrated a statistically significant improvement in nutrition knowledge (p < .001). Ancillary analysis revealed that participants with the lowest quartile of pre-program nutrition knowledge had the greatest improvements in their post-GNKQ-R scores (Mean change 16.4 ± 13.9). Questionnaire data revealed participants' subjective knowledge changes and their feedback on the nutrition component of the program.

CONCLUSIONS

The eight-week MENTOR program effectively improved nutrition knowledge among people with physical disabilities. This demonstrates the impact the program can have on individual knowledge levels. Additionally, these results support the implementation of the program to the broader population of people with disabilities.

Equity-related 'knots' in theory of change development: Conceptualization and case illustrations

Theory of change (ToC) is an approach widely used to guide planning, implementing, and evaluating change initiatives. While there is substantial guidance, there has been little attention on equity within ToC research and practice. We propose and illustrate the metaphor of 'knots' to frame practical and ethical challenges that arise when centering equity within ToC processes. Drawing on our experiences using a ToC approach in two case examples, we identify and illustrate five equity-related knots: (a) clarify root causes, pathways, and success; (b) facilitate participation across power and perspective differences; (c) integrate research evidence and practitioner knowledge; (d) represent complex change visually; and (e) creatively navigate constraints. We show why framing these as knots can help practitioners make wise judgments within the circumstances and close with recommendations for including knots in ToC processes, reporting, and guidance.

Evaluating the impact of using storytelling and visual multimedia music intervention to improve knowledge of the Child Rights Act and reduce the propensity to engage in child marriage

Child marriage has continued to rear its ugly head in Nigerian society. This study aimed to evaluate the impact of storytelling and multimedia music interventions in improving knowledge of the Child Rights Act and reducing the propensity to engage in child marriage. The researchers applied a quasi-experimental design and collected data using a structured questionnaire. The children were assigned into three groups (control, storytelling and multimedia music) of 173 participants. It was found that the interventions were effective. In particular, while storytelling contributed more to reducing the propensity to engage in child marriage, multimedia music contributed more to improving knowledge of the Child Rights Act. These results suggest that storytelling and multimedia music interventions can be effective approaches for addressing the lingering problem of child marriage in Nigeria.

Participatory training evaluation: Steps from the Center for Native American Health Native-CHART symposium

This paper presents a case example of the Native-CHART Training Evaluation and describes the process of planning and administering a paper evaluation during the Native-CHART symposium in November 2019 led by the Center for Native American Health (CNAH) and an external evaluator. Training evaluation methodologies and the data collection instrument were grounded in the Health Belief Model (HBM) where health-related chronic disease and risk factor knowledge translates to perceived susceptibility, benefits, barriers, and self-efficacy. Kirkpatrick's Four-level Training Evaluation Model explored learning, reaction, behaviors, and results. The evaluation aims centered around the following questions: 1)Who attended the symposium, and why did they attend? 2)What knowledge did participants gain at the symposium? 3)Will attendees change their behaviors as a result of attending the symposium? 4) What parts of the symposium were most valuable? And 5) How can the symposium be improved? Data collected at the symposium answered these questions. After the Native-CHART symposium, CNAH staff and the external evaluator met to reflect on the steps necessary to plan and implement a participatory training evaluation. From these discussions, eight steps emerged. This paper presents these steps along with recommendations for future work. Participatory and collaborative approaches in training evaluation and the steps included in this case example may be useful to evaluators, communities, and programs working on designing and evaluating various trainings with Tribal populations.

Evaluation of the concerns of Spanish university students in the face of current major challenges

The concerns of students and young people have been on the rise in recent years. Recent events such as the Covid-19 pandemic or the invasion of Ukraine have contributed to this. Mental health, the high cost of living, the need to re-evaluate the way we work or the fight against climate change are among the main concerns of young people. The aim of this work is to find out the level of concern of university students about these problems that can condition their current life and to analyze whether these concerns can affect their academic performance, also distinguishing according to gender and age. To this end, a nationwide survey was carried out among this group in Spain. This work is the first of its kind to be carried out in Spain. The results obtained may be of great use to both educational institutions and government agents, with a view to offering policies that will improve the educational level and welfare of university students.

Using personal network analysis to understand the interaction between programmes' facilitators and teachers in psychoeducational interventions

In this study we use personal network analysis to examine the social relationships taking place throughout programme implementation. Previous literature on psychosocial intervention has used network analysis techniques to examine: (a) the interaction between participants, (b) the facilitators' link to the intervention target group, (c) the transfer of knowledge between experts and facilitators, and (d) the interaction of facilitators with each other. However, there has been little research on how facilitators connect with other figures in their organisational context, impacting both intervention fidelity and the fit of the programme to the local context. In this study we combine the analysis of personal networks with qualitative interviews with 102 teachers in 72 schools in Barranquilla (Colombia), with whom we describe the implementation of the psychoeducational programmes Pisotón and Metodologías Flexibles. The results show that programme implementation networks not only rely on facilitators but also on the contribution of the schools' director of studies, the coupling with the regular classroom teachers, and occasional collaborations from the rest of the school staff. After conducting a cluster analysis, we detected the existence of two types of personal networks, some based on the functioning of highly cohesive teams and others with a greater level of centralisation around the head of studies. Implementation networks not only enable the implementation of programme activities, but also the integration of the programme into educational organisations. In the discussion we reflect on how the analysis of facilitators' personal networks can be used to improve the process of programme implementation.

Evaluation of the public policy impacts on Monkeypox in Brazil

Brazil ranked third in the number of Monkeypox infected worldwide in early September 2022 and eighth in multiple deaths. Brazilian Ministry of Health prepared a public policy to face the smallpox outbreak. This paper aims to analyze the governmental public policy' impacts on Monkeypox using survival analysis. The information in the database was collected from epidemiological bulletins on the official websites of the Brazilian Ministry of Health and the World Health Organization (WHO). The survival analysis with parametric statistical analysis, semiparametric with Cox regression, and nonparametric analysis were used. The inference of causality was perceived by the impact caused by the national public policy with the proportional reduction in the number of cases in the treatment group (Chi-sq = 117.783, p < 0.001), contrary to what was perceived in the control group, as well as survival about the time of the states that elaborated their plans based on what was made available by the government. The need to evaluate government projects should be within the scope of project management in Brazilian states and provide for more assertive decision-making in the fight against smallpox.

Comparison of industrial engineering programs in Colombia based on standardized test results

This study aims to evaluate the performance of industrial engineering programs across various Higher Education Institutions (HEIs) in Colombia, focusing on the SaberPRO test outcomes from 2016 to 2018. The evaluation considers three primary dimensions: the type of funding (public vs. private HEIs), geographical region, and program accreditation status. The methodology involved gathering and analyzing Saber Pro test results from students enrolled in industrial engineering programs at 82 Colombian HEIs during the specified period. In Colombia, the SaberPRO Exam, administered by the Colombian Institute for the Evaluation of Education (ICFES), is a critical measure for assessing the quality of future industrial engineering graduates. This research employs an analytical framework that includes Analysis of Variance (ANOVA) and hypothesis testing for mean comparisons to scrutinize the performance of these programs. Findings reveal notable disparities in performance between public and private institutions and between accredited and non-accredited programs. Additionally, through Multiple Factor Analysis (MFA), the study identifies the competencies with the highest discriminative capacity, recognizing 15 exemplary programs out of the 82 evaluated.