Improving the Usefulness and Use of Meta-Analysis to Inform Policy and Practice

This chapter begins with an overview of recent developments that have encouraged and facilitated greater use of research syntheses, including Meta-Analysis, to guide public policy and practice in education, workforce development, and social services. It discusses the role of Meta-Analysis for improving knowledge of the effectiveness of programs, policies, and practices and the applicability and generalizability of that knowledge to conditions other than those represented by the study samples and settings. The chapter concludes with recommendations for improving the potential of Meta-Analysis to accelerate knowledge development through changing how we design, conduct, and report findings of individual studies to maximize their usefulness in Meta-Analysis as well as how we produce and report Meta-Analysis findings. The paper includes references to resources supporting the recommendations.

How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

Mixed methods evaluation of a jail diversion program: Impact on arrests and functioning

This mixed methods study had two aims: (1) to examine the effectiveness of a jail diversion program in reducing recidivism and promoting educational and employment outcomes; and (2) to qualitatively explore mechanisms through which the program was effective. Participants were 17 individuals arrested for drug offenses who participated in an intensive, law enforcement-based jail diversion program, and 17 individuals in a comparison group. Arrests were extracted from police records, and education and employment were extracted from program data. Four intervention participants completed qualitative interviews. Arrest rates in the intervention group decreased significantly postintervention, and arrest rates in the intervention group were numerically lower than those in the comparison group. Participants experienced significant increases in employment and driver's license status. Participants also identified mechanisms through which the program was effective. This jail diversion program shows promise in reducing recidivism and promoting adaptive functioning. Jail diversion programs that include mentorship, peer support, and removal of barriers to success may be particularly effective.

Ageism webinars: a needs-based continuing education training for the national aging network

To combat ageism, the National Resource Center on Nutrition and Aging (NRCNA) launched two Ageism Webinars (AWs). The study's aims were to 1) assess the effectiveness of the AWs for increasing participants' perceived familiarity (FAM) and knowledge (KNOW) and producing high behavioral intent to implement recommended action steps and quality among participants, and 2) practice equitable evaluation by ascertaining whether outcomes were fairly achieved by participants regardless of race, age, and education. A convenience sample of mostly educated non-Hispanic females with an average age of 52 years (n = 193) completed a retrospective online survey post-webinar. A retrospective Likert scale noted an overall increase in perceived FAM and KNOW (p < 0.001). However, these changes were significantly smaller (p < 0.05) among those from historically marginalized races and ethnicities. The theory of planned behavior (TPB) items predicted high intention to complete the recommended action step following the webinars. High satisfaction for webinars was reported, with 98.7% wanting to attend future NRCNA training. This study provides evidence of an effective training modality for addressing ageist perspectives, and the demonstrated differences provide insight into how to improve the AWs and the evaluation of future webinars.

Cost-effectiveness of One Health interventions for rabies elimination: a systematic review

The 'One Health' (OH) approach is the most promising idea in realising the global goal of eliminating canine-mediated human rabies by 2030. However, taking an OH approach to rabies elimination can mean many different things to different people. We conducted a systematic review scrutinizing economic evaluations (EEs) retrieved from MEDLINE OVID, Embase OVID, Global Health OVID, CINAHL EBSCO and ECONLIT EBSCO that used the OH approach with the intent of identifying cost-effective sets of interventions that can be combined to implement an optimal OH-based rabies elimination program and highlight key gaps in the knowledge base. Our review suggests that an optimal OH program to tackle rabies should incorporate mass dog vaccination and integrated bite case management in combination with efficient use of post-exposure prophylaxis along with a shift to a 1-week abbreviated intradermal rabies vaccine regimen in humans. We recommend that future EEs of OH interventions for rabies elimination should be performed alongside implementation research to ensure proposed interventions are feasible and adopt a wider societal perspective taking into account costs and outcomes across both the human health and animal welfare sectors. The systematic review has been registered with PROSPERO.

Effectiveness of targeted intervention program under the national AIDS control program among Hijra and transgender population: Evidence from Integrated Biological and Behavioral Surveillance, 2014-15

INTRODUCTION

The hijra and transgender (H/TG) population in India is vulnerable to HIV/AIDS. India had instituted a targeted intervention (TI) program to reduce this vulnerability. We aimed to measure the effectiveness of the TI program for H/TG.

MATERIALS AND METHODS

The National Integrated Behavioral and Biological Survey (IBBS) was carried out in 2014-15. H/TG data from IBBS was analyzed. Bivariate and multivariate logistic regression were used to calculate the unadjusted and adjusted odds ratios with 95% confidence interval. Condom use during the last sexual intercourse, and the consistent condom use in the last one month were considered as indicators of program effectiveness. The Propensity Score Matching (PSM) method was used to assess the effectiveness.

RESULTS

We found that the participants who had received condoms from peer educator/outreach worker were 1.74 and 1.40 times more likely to use condoms in the last sexual intercourse (aOR: 1.74, CI: 1.35 - 2.26) and consistent condom use in the last one month (aOR: 1.40, CI: 1.12 - 1.74) respectively compared to the participants who did not receive the condom. The matched-samples estimate (i.e., average treatment effect on treated) for the condom use during the last sexual intercourse increased by 13.0%, i.e., 0.13 (CI; 0.08 - 0.18) and consistent condom use in the last one month increased by 5.0%, i.e., 0.05 (CI; 0.00 - 0.10) among those who had received condoms from the peer educator/outreach worker compared with those who had not received condom, respectively.

CONCLUSIONS

The TI program intervention for H/TG was effective in reducing HIV risk behavior as evidenced by increase in use of condom during last sexual intercourse, and consistent condom use in the last one month.

Perspective: Leveraging Electronic Health Record Data Within Food Is Medicine Program Evaluation: Considerations and Potential Paths Forward

Government, health care systems and payers, philanthropic entities, advocacy groups, nonprofit organizations, community groups, and for-profit companies are presently making the case for Food is Medicine (FIM) nutrition programs to become reimbursable within health care services. FIM researchers are working urgently to build evidence for FIM programs' cost-effectiveness by showing improvements in health outcomes and health care utilization. However, primary collection of this data is costly, difficult to implement, and burdensome to participants. Electronic health records (EHRs) offer a promising alternative to primary data collection because they provide already-collected information from existing clinical care. A few FIM studies have leveraged EHRs to demonstrate positive impacts on biomarkers or health care utilization, but many FIM studies run into insurmountable difficulties in their attempts to use EHRs. The authors of this commentary serve as evaluators and/or technical assistance providers with the United States Department of Agriculture's Gus Schumacher Nutrition Incentive Program National Training, Technical Assistance, Evaluation, and Information Center. They work closely with over 100 Gus Schumacher Nutrition Incentive Program Produce Prescription FIM projects, which, as of 2023, span 34 US states and territories. In this commentary, we describe recurring challenges related to using EHRs in FIM evaluation, particularly in relation to biomarkers and health care utilization. We also outline potential opportunities and reasonable expectations for what can be learned from EHR data and describe other (non-EHR) data sources to consider for evaluation of long-term health outcomes and health care utilization. Large integrated health systems may be best positioned to use their own data to examine outcomes of interest to the broader field.

An evaluation of an employment assistance program focused on people living with HIV in Toronto, Canada

Unemployment is more common among people living with HIV (PLWH) compared to the general population. PLWH who are employed have better physical and mental health outcomes compared to unemployed PLWH. The main objective of this mixed-methods study was to conduct a program evaluation of Employment Action (EACT), a community-based program that assists PLWH in Toronto, Ontario, Canada to maintain meaningful employment. We extracted quantitative data from two HIV services databases used by EACT, and collected qualitative data from 12 individuals who had been placed into paid employment through EACT. From 131 clients included in the analysis, 38.1% (n = 50) maintained their job for at least 6 weeks within the first year of enrollment in the EACT program. Gender, ethnicity, age, and first language did not predict employment maintenance. Our interviews highlighted the barriers and facilitators to effective service delivery. Key recommendations include implementing skills training, embedding PLWH as EACT staff, and following up with clients once they gain employment. Investment in social programs such as EACT are essential for strengthening their data collection capacity, active outreach to service users, and sufficient planning for the evaluation phase prior to program implementation.

Collaborative sustainability planning for an outcome measure in outpatient stroke rehabilitation: A qualitative description study

RATIONALE

Evidence suggests that sustainability planning and the use of a collaborative approach to planning result in better sustainability outcomes and more relevant knowledge. Yet, both approaches appear to be underutilized. A detailed description of collaborative sustainability planning may encourage the use of these two impactful strategies.

AIMS AND OBJECTIVES

To explore the collaborative sustainability planning process for a single outcome measure in three rehabilitation sites.

METHODOLOGY

Within the Mayo-Portland Adaptability Inventory-version 4 (MPAI-4) implementation project, we conducted a qualitative description study. We used data from 12 core sustainability planning meetings and 108 follow-up meetings that included a total of 31 clinical and research team participants. Sustainability planning was informed by an MPAI-4-specific implementation guide, and by the results from a realist review of the sustainability of rehabilitation practices and the Clinical Sustainability Assessment Tool. We analyzed qualitative data using thematic content analysis.

RESULTS

Three themes describe the collaborative sustainability planning process: (1) "collaboration as a driver for sustainability" which captures the active collaboration underpinning sustainability planning; (2) "co-creation of a sustainability plan to achieve shared objectives" which captures the identified barriers and facilitators, and selected sustainability strategies linked to one of six collaboratively identified shared objectives; and (3) "the iterative nature of sustainability planning" which captures the necessity of an agile and responsive sustainability planning process.

CONCLUSION

Identified strategies may be useful to support (collaborative) sustainment. Future research could investigate the effect of collaborative sustainability planning on sustainability objectives, and the relationship between these objectives.

e-Delphi in the outdoors: Stakeholder contributions to the development of a wellbeing-focused outdoor adventure education intervention program

ISSUE ADDRESSED

Outdoor adventure education (OAE) (programs involving outdoor activities such as rock climbing or white-water canoeing) that participants perceive as risky, conducted in a social support setting, can be utilised by practitioners to elicit changes in educational and psychosocial outcomes to support participant adolescent wellbeing.

METHODS

This study garnered the opinions of an expert OAE panel on the content of future programs aiming to impact adolescent wellbeing. The panel consisted of local (Western Australia, n = 7), national (Australia, n = 4), and international (Canada, Germany, New Zealand, United Kingdom, United States, n = 7) experts. A two-round, mixed-methods Delphi approach was employed. Extensive formative work led to the development of a series of open-ended questions requiring qualitative responses for round one. Panellists were also asked to respond to 17 statements using Likert scales in the second round.

RESULTS

After analysis, a consensus was reached for all statements, with five statements having high consensus and being considered important by panellists.

CONCLUSIONS

The statement 'Equity for all participants requires flexible delivery and facilitation' had the highest level of agreement amongst panellists. Connections, authentic experiences, and equitable experiences developed as key themes. SO WHAT?: Future OAE interventions focused on wellbeing impact could use the findings of this research as a basis for program design.

Pilot evaluation of a behavioral weight loss program for adults with physical disabilities: State of Slim Everybody usability and feasibility

Background

People with physical disabilities (PWD) have a higher prevalence of obesity than populations without disability, but most evidence-based weight loss programs have not included this population. The State of Slim (SOS) program is an evidence-based weight loss program that has demonstrated success in producing weight loss in populations without disability, but it has not been adapted for or evaluated in PWD.

Methods

The SOS program was systematically adapted using the evidence-informed Guidelines, Recommendations, and Adaptations Including Disability (GRAIDs) framework. A total of 35 participants enrolled in the State of Slim Everybody program. The program was offered entirely online. Body weight, attendance, and food log completion were also tracked weekly. The program length was 16.5 h and included weekly group instruction, with optional one-on-one sessions provided upon request. Following completion, participants completed post-evaluation surveys on overall satisfaction with the program. The primary outcomes were program effectiveness (i.e., body weight), usability, and feasibility.

Results

Thirty-two out of 35 participants completed the program, representing a retention rate of 91.4%. Average weight loss was 10.9% (9.9 ± 0.7 kg (t (31) = -13.3, p =< 0.0001)). On a 1 (dissatisfied/completely useless) to 5 (very satisfied/completely helpful) Likert scale, the average score for overall program satisfaction was 4.8 ± 0.1 and program helpfulness 4.6 ± 0.1.

Conclusion

The State of Slim Everybody program demonstrated significant weight loss and good usability and feasibility in PWD. Existing adaptation frameworks can be used to create inclusive health promotion programs for adults with physical disabilities.

UK general population's willingness to pay for dental check-ups

OBJECTIVES

Choices about which dental treatments to provide in a publicly funded system should be guided by the value that the general population place on those treatments. The aim of this study was to estimate United Kingdom (UK) general population willingness to pay (WTP) for dental check-ups, and to investigate what factors influence WTP.

METHODS

WTP was elicited using a hypothetical question in an online survey. The sample consisted of 594 participants, nationally representative of the UK general population in terms of age and gender. Regression analysis was used to examine what factors are associated with WTP. Analyses were conducted including and excluding protest answers.

RESULTS

The mean WTP for a dental check-up is £31.32 for the full sample and £33.17 excluding protest answers. Respondents on higher incomes and those with higher university education had higher WTP. Respondents in Scotland were WTP less than respondents living in the rest of the UK which may be the result of NHS dental check-ups being free to patients in Scotland.

CONCLUSION

The general UK population value dental check-ups. This study provides estimates of WTP for dental check-ups which can be used in Cost-Benefit Analyses.

The competency index for clinical research professionals: a potential tool for competency-based clinical research academic program evaluation

Background

Accreditation of graduate academic programs in clinical research requires demonstration of program achievement of Joint Task Force for Clinical Trial Competence-based standards. Evaluation of graduate programs include enrollment, student grades, skills-based outcomes, and completion rates, in addition to other measures. Standardized measures of competence would be useful.

Methods

We used the Competency Index for Clinical Research Professionals (CICRP), in a separate-sample pretest-posttest study to measure self-confidence or self-efficacy in clinical research competency comparing cohorts of students entering and completing a master's degree program in clinical research across three semesters (summer 2021 - spring 2022). CICRP is a 20-item Likert scale questionnaire (0 = Not at all confident; 10 = extremely confident).

Results

The study sample of 110 students (54 in the entry course, 56 in the exit course) showed overall 80.9% entered the program with only a baccalaureate degree and 55.5% had no prior experience in managing clinical trial research. Cronbach alpha for the instrument showed a high level of content validity (range 0.93-0.98). Median CICRP item rating range at entry was [1, 6] and at exit [7, 10]. Mean CICRP total score (sum of 20 items) at entry was 72.7 (SD 41.9) vs. 167.0 (SD 21.1) at exit (p < 0.001). Mean total score at program entry increased with increasing years of clinical trial management experience but attenuated at program exit.

Conclusion

This is the first use of the CICRP for academic program evaluation. The CICRP may be a useful tool for competency-based academic program evaluation, in addition to other measures of program excellence.

Implementation of Cognitive Health Services in Large Systems of Care: Highlights From Coordinated Specialty Care for First Episode Psychosis

BACKGROUND AND HYPOTHESIS

With increasing recognition of the importance of cognitive health for recovery in people with psychosis, questions arise as to how to implement cognitive health services in large systems of care. This paper describes the implementation of cognitive health services in OnTrackNY (OTNY), a network of clinics delivering a Coordinated Specialty Care treatment model for early psychosis, with the goal of documenting the processes, challenges, and useful adaptations.

STUDY DESIGN

In 2018, OTNY piloted a Cognitive Health Toolkit for implementation across 18 affiliated clinics. The toolkit intended to identify the cognitive health needs of individuals early in the course of psychosis and to integrate cognitive health into the vocabulary of wellness and recovery. Implementation involved creating mechanisms for staff training and support to, in turn, help participants improve how they use cognitive skills in daily life.

STUDY RESULTS

The toolkit was disseminated to all 28 OTNY programs throughout New York state by 2023. When simple assessment and decision-making tools were embedded in routine care practices, the majority of participants identified that improving memory, attention, and critical thinking skills would be helpful. Consistently, about 70% of those asked wanted to learn more about how to better their cognitive health.

CONCLUSIONS

Cognitive health services can be implemented in large systems of care that provide a multi-level system of implementation supports. Organizational facilitators of implementation include a training program to educate about cognitive health and the delivery of cognitive health interventions, and embedded quality assurance monitoring and improvement activities.

Evaluation of a Fall Prevention Program to Reduce Fall Risk and Fear of Falling Among Community-Dwelling Older Adults and Adults with Disabilities

Purpose

The overarching goal of the program evaluation was to determine the reach and impact of the District-funded Safe At Home (SAH) modification program in reducing falls, fall injuries, and fear of falls among community-dwelling older adults and adults with disabilities. The SAH program has served over 6000 adults since 2016, the majority of whom are women (79%) and over age 60 (92%).

Materials and Methods

Letters were mailed in September 2022 to clients (n=492) who had home modifications completed between October 2021-March 2022 inviting them to participate in a brief phone survey about program satisfaction, falls, fall location, and severity. The validated Fall Efficacy Scale (FES) was administered pre (at first visit), post (at last visit), and during the phone survey (within 6 months to 1 year of program completion) to assess fear of falling. The response rate was 55% (n=241).

Results

Older adults (n=219) and adults with disabilities (n=22) reported high program satisfaction. Most clients, 79%, did not report a fall since the completion of the home modifications. The majority of falls reported, 76%, occurred inside the home. The average evaluation FES score was 32.5 (SD=22.6, range 10-100), indicating relatively low fear of falling. Higher FES scores were associated with a greater likelihood of reporting a fall (r=0.44, p < 0.001, n=51) and older age (r = 0.17, p < 0.01). FES scores were not related to gender. Evaluation FES scores were significantly lower than the pre-FES scores, indicating a reduction in fear of falling and positive impact of the home modifications (T(107) = 5.14, p < 0.001).

Conclusion

The client-centered SAH program demonstrates significant reductions in falls, fear of falling, and high satisfaction among clients. Recommendations include program expansion to offer other evidence-based components to reduce falls and support safe aging in place.

Mortality Among HIV-Infected Adults on Antiretroviral Therapy in Southern Uganda

BACKGROUND

Monitoring and evaluation of clinical programs requires assessing patient outcomes. Numerous challenges complicate these efforts, the most insidious of which is loss to follow-up (LTFU). LTFU is a composite outcome, including individuals out of care, undocumented transfers, and unreported deaths. Incorporation of vital status information from routine patient outreach may improve the mortality estimates for those LTFU.

SETTINGS

We analyzed routinely collected clinical and patient tracing data for individuals (15 years or older) initiating antiretroviral treatment between January 2014 and December 2018 at 2 public HIV care clinics in greater Rakai, Uganda.

METHODS

We derived unadjusted mortality estimates using Kaplan-Meier methods. Estimates, adjusted for unreported deaths, applied weighting through the Frangakis and Rubin method to represent outcomes among LTFU patients who were successfully traced and for whom vital status was ascertained. Confidence intervals were determined through bootstrap methods.

RESULTS

Of 1969 patients with median age at antiretroviral treatment initiation of 31 years (interquartile range: 25-38), 1126 (57.2%) were female patients and 808 (41%) were lost. Of the lost patients, 640 patient files (79.2%) were found and reviewed, of which 204 (31.8%) had a tracing attempt. Within the electronic health records of the program, 28 deaths were identified with an estimated unadjusted mortality 1 year after antiretroviral treatment initiation of 2.5% (95% CI: 1.8% to 3.3%). Using chart review and patient tracing data, an additional 24 deaths (total 52) were discovered with an adjusted 1-year mortality of 3.8% (95% CI: 2.6% to 5.0%).

CONCLUSIONS

Data from routine outreach efforts by HIV care and treatment programs can be used to support plausible adjustments to estimates of client mortality. Mortality estimates without active ascertainment of vital status of LTFU patients may significantly underestimate program mortality.

Twelve tips for Natural Language Processing in medical education program evaluation

With the increasing application of Natural Language Processing (NLP) in Medicine at large, medical educators are urged to gain an understanding and implement NLP techniques within their own education programs to improve the workflow and make significant and rapid improvements in their programs. This paper aims to provide twelve essential tips inclusive of both conceptual and technical factors to facilitate the successful integration of NLP in medical education program evaluation. These twelve tips range from advising on various stages of planning the evaluation process, considerations for data collection, and reflections on preprocessing of data in preparation for analysis and interpretation of results. Using these twelve tips as a framework, medical researchers, educators, and administrators will have an understanding and reference to navigating applications of NLP and be able to unlock its potential for enhancing the evaluation of their own medical education programs.

Unlocking the mechanisms of change in the MAMAACT intervention to reduce ethnic disparity in stillbirth and newborns' health: integration of evaluation findings

Ethnic disparities in stillbirth exist in Europe and suboptimal care due to miscommunication is one contributing cause. The MAMAACT intervention aimed to reduce ethnic disparity in stillbirth and newborns' health through improved management of pregnancy complications. The intervention encompassed training of antenatal care midwives in cultural competencies and intercultural communication combined with health education materials for the expecting parents about symptoms of pregnancy complications. The evaluation consisted of a qualitative in-depth implementation analysis and a process evaluation embedded in a cluster randomized trial including 19 of 20 maternity wards in Denmark. In this article, the findings from the different evaluation perspectives are integrated. The integration follows the principles of realist evaluation by analyzing to what extent the MAMAACT activities were generating mechanisms of change in interaction with the context. The integration analysis shows that the health education materials in the MAMAACT intervention contributed to heightened health literacy concerning pregnancy complications among pregnant women. Additionally, the training of midwives in cultural competency and intercultural communication raised awareness among midwives. Nonetheless, the exclusive emphasis on midwives and the inflexibility in care provision hindered them from changing their communication practices. To enhance the cultural competence in maternity care, it is essential to implement more comprehensive initiatives involving healthcare professionals in maternity care at all levels, from pregraduate to postgraduate. Adequate interpreter services and management support should also be ensured. Currently, the Danish antenatal care system faces challenges including inadequate information transfer between healthcare sectors, insufficient differentiation of care, and inflexibility in midwife scheduling. This results in a lack of responsiveness to the individual needs of women with immigrant backgrounds, potentially reproducing health inequities.

Diffusion of excellence: evaluating a system to identify, replicate, and spread promising innovative practices across the Veterans health administration

Introduction

The Veterans Health Administration (VHA) Diffusion of Excellence (DoE) program provides a system to identify, replicate, and spread promising practices across the largest integrated healthcare system in the United States. DoE identifies innovations that have been successfully implemented in the VHA through a Shark Tank style competition. VHA facility and regional directors bid resources needed to replicate promising practices. Winning facilities/regions receive external facilitation to aid in replication/implementation over the course of a year. DoE staff then support diffusion of successful practices across the nationwide VHA.

Methods

Organized around the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework, we summarize results of an ongoing long-term mixed-methods implementation evaluation of DoE. Data sources include: Shark Tank application and bid details, tracking practice adoptions through a Diffusion Marketplace, characteristics of VHA facilities, focus groups with Shark Tank bidders, structured observations of DoE events, surveys of DoE program participants, and semi-structured interviews of national VHA program office leaders, VHA healthcare system/facility executives, practice developers, implementation teams and facilitators.

Results

In the first eight Shark Tanks (2016-2022), 3,280 Shark Tank applications were submitted; 88 were designated DoE Promising Practices (i.e., practices receive facilitated replication). DoE has effectively spread practices across the VHA, with 1,440 documented instances of adoption/replication of practices across the VHA. This includes 180 adoptions/replications in facilities located in rural areas. Leadership decisions to adopt innovations are often based on big picture considerations such as constituency support and linkage to organizational goals. DoE Promising Practices that have the greatest national spread have been successfully replicated at new sites during the facilitated replication process, have close partnerships with VHA national program offices, and tend to be less expensive to implement. Two indicators of sustainment indicate that 56 of the 88 Promising Practices are still being diffused across the VHA; 56% of facilities originally replicating the practices have sustained them, even up to 6 years after the first Shark Tank.

Conclusion

DoE has developed a sustainable process for the identification, replication, and spread of promising practices as part of a learning health system committed to providing equitable access to high quality care.

Teaching happiness to students - implementation and evaluation of a program aiming at promoting wellbeing in elementary schools

Since the beginning of the COVID-19 pandemic, the prevalence of mental disorders in children and adolescents has increased significantly. Evidence shows that childhood mental disorders can have serious consequences on psychosocial, cognitive, and physical development. Approaches from Positive Education go further than the urgently needed prevention of mental disorders by aiming directly at promoting subjective, psychological, and social wellbeing. The present study describes the implementation of a brief program to promote wellbeing in 15 elementary schools. For this purpose, in a regular university seminar, students of teaching and educational science were instructed to give 11 "happiness lessons" for fourth graders in a team of two and in the presence of the class teacher over the course of 3 months. Quantitative data were collected from children and parents in the treatment group classes and in the parallel classes serving as the waiting control group at four measurement points (pre, post, 1- and 2-month follow-up). We assessed psychological wellbeing, negative emotions and moods, parent support and home life, perception of the school environment, and self-esteem of the children with established instruments with versions for children and their parents and the frequency of positive and negative emotions of the children in self-report only. Additionally, we applied ad hoc items on subjective perception of the project and open questions in the treatment group. Data were analyzed with EffectLiteR using multigroup structural equation models. Results showed a small significant effect for negative emotions with the children's data and a medium effect for psychological wellbeing in the perception of the parents at the 1-month follow-up. Interaction effects suggest that lower baseline levels in parent support and home life and self-esteem would increase the treatment effect for these constructs. The need for more grounded framework in positive education and the inclusion of more qualitative methods as well as suggestions to improve the program in the sense of a whole school approach are discussed.

Adjusting for race in metrics of organ procurement organization performance

The Scientific Registry of Transplant Recipients has previously reported the effects of adjusting for demographic variables, including race, in the Centers for Medicare & Medicaid Services (CMS) organ procurement organization (OPO) performance metrics: donation rate and transplant rate. CMS chose not to adjust for most demographic variables other than age (for the transplant rate), arguing that there is no biological reason that these variables would affect the organ donation/utilization decision. However, organ donation is a process based on altruism and trust, not a simple biological phenomenon. Focusing only on biological impacts on health ignores other pathways through which demographic factors can influence OPO outcomes. In this study, we update analyses of demographic adjustment on the OPO metrics for 2020 with a specific focus on adjusting for race. We find that adjusting for race would lead to 8 OPOs changing their CMS tier rankings, including 2 OPOs that actually overperform the national rate among non-White donors improving from a tier 3 ranking (facing decertification without possibility of recompeting) to a tier 2 ranking (allowing the possibility of recompeting). Incorporation of stratified and risk-adjusted metrics in public reporting of OPO performance could help OPOs identify areas for improvement within specific demographic categories.

Impact of Safe@Home on Placement and Permanency Outcomes: Results of a Quasi-Experimental Study

This study assessed the effectiveness of Safe@Home, an in-home intervention to (1) prevent out-of-home placement for children at imminent risk of placement, and (2) minimize time in out-of-home care for children already in foster care. Using Coarsened Exact Matching, children who received Safe@Home were matched to a comparison group of children served before Safe@Home was available in their community. All children were determined by the child welfare agency to be unsafe and in need of immediate intervention. The matched samples (Safe@Home n = 510, Comparison n = 851) showed strong baseline equivalence on child age, race/ethnicity, previous history of child welfare involvement, and safety threats. Children who received Safe@Home experienced lower rate of out-of-home placements, higher rate of permanency with a parent (sustained for 12 months after the end of Safe@Home), fewer days in out-of-home care, and shorter time to case closure relative to children who received treatment as usual. There was no effect of Safe@Home on post-permanency outcomes of maltreatment and entry or re-entry after case closure.

Microlearning: Evidence-based education that is effective for busy professionals and short attention spans

Microlearning uses short educational interventions to provide learners with the necessary knowledge and skills to perform specific tasks or solve immediate problems. This approach is increasingly used across digital platforms to engage learners and foster quick comprehension. Microlearning can be used in clinical genetics education to deliver a comprehensive educational intervention that is segmented into smaller discrete but complimentary components. This report discusses one group's approach to using microlearning in clinician education and provides tips that can be applied to other educational efforts. High-quality genetics education has the potential to be disseminated across multiple delivery methods and to multiple audiences, thereby increasing its impact and reach.

Evaluation of a Palliative Care Longitudinal Curriculum for Medical Students Using the Context-Input-Process-Product Model

Palliative care (PC) longitudinal curricula are increasingly being recognized as important in Undergraduate Medical Education (UME). They are however, not yet commonplace, and where they do exist may be implemented without a systematic, prospective approach to curriculum evaluation. This paper describes an implementation of a new longitudinal curriculum at the Perelman School of Medicine (PSOM) at the University of Pennsylvania. We used the Context Input Process Product (CIPP) model, a holistic evaluation model, to assess the local environment, design the curriculum, guide the improvement process, and evaluate outcomes. Comprehensive models such as CIPP provide a more robust approach to curriculum evaluation than outcomes-only models and may be of use to other programs who are implementing new curricula or improving upon existing programs.

A team-based approach to cardiogenomic education

While many genetic professionals are involved in the education of lay and professional audiences, most do not have formal training in education theory and program design. Partnerships with adult education experts can provide additional resources and improve the level of instruction, thereby increasing the impact of an educational intervention. This report discusses the experience of a multidisciplinary team of educators, clinicians, and researchers partnering to develop evidence-based education for cardiology practitioners. It includes practical advice for how clinicians and educators can develop more effective education through collaboration, needs assessment, instructional design, and iterative content development.

Proposed use of entrustable professional activities (EPAs) in genetic counseling for clinical training and assessment

We introduce Entrustable Professional Activities (EPAs) as a potential framework for clinical training and assessment in genetic counseling. We discuss advantages of this approach, review how EPAs complement Practice-Based Competencies (PBCs), describe our process of generating proposed "core" EPAs, provide examples of specialty-specific EPAs, discuss the concept of entrustment in clinical training, and propose an approach to implementation.

Genetic counseling program remediation practices for students underperforming in clinical skills: An exploratory study

Program-level clinical remediation in genetic counseling training programs aims to help students who are underperforming gain clinical skills to successfully manage clinical counseling sessions with patients. Student remediation often requires intervention, including discussions with program leadership and/or a formal remediation plan through the program. This study surveyed genetic counseling program leaders to explore the remediation landscape by identifying skills in which students underperformed, program remediation activities to improve skills, and remediation outcomes. Thirteen participants indicated their program required at least one student to complete program-level clinical remediation during the last 10 years. Eight of the 13 programs (61.5%) required at least one student to participate in clinical remediation for underperformance in professionalism, seven (53.8%) for underperformance in educating patients, six (46.2%) for underperformance in critical thinking, and two (15.4%) for underperformance in demonstrating empathy. Nineteen students were remediated for underperformance in critical thinking. Of those 19 students, one student (5.2%) was dismissed from the training program, and five students (26.3%) chose to withdraw from their program. One of 13 (7.7%) students remediated for underperformance in educating patients and one of 11 (9.1%) students remediated for underperformance in professionalism chose to withdraw from their programs. All students remediated for underperformance in demonstrating empathy successfully completed program-level clinical remediation and graduated. The most frequently endorsed factor positively associated with remediation success was completion of additional in-person patient encounters. The most frequent barrier was a student's poor mental health. Participants most frequently endorsed identification of resources for specific areas of remediation to improve their programs' efficacy in clinical remediation practices. This exploratory study provides valuable information describing clinical skills that require remediation in genetic counseling graduate training, the remediation practices utilized by training programs, and resources that may increase remediation success.

A community-based, interdisciplinary approach to improving HPV vaccine uptake at a large urban public university

The purpose of this project was to pilot and determine the outcomes of a multi-faceted, interdisciplinary human papillomavirus awareness and vaccination campaign within a university community.

Students, faculty, and staff of a large urban Midwest-based public university.

METHODS

This consisted of five key interventions: promotion and outreach, volunteer training, patient education, access, and documentation. Perceived knowledge and attitudes toward HPV and the HPV vaccine, and HPV vaccine uptake were measured as outcome variables.

The project resulted in statistically significant increases in perceived knowledge regarding HPV and the HPV vaccine, willingness to receive HPV vaccination, and likeliness to recommend HPV vaccination (p < 0.001). The project also resulted in a 76.6% increase in the number of HPV vaccines administered at the university's Campus Health Center.

A novel HPV awareness and vaccination campaign led to increases in knowledge regarding HPV and HPV vaccines, and substantially increased HPV vaccinations administered at a large, urban public university.

Examining alignment of community health teams' preferences for health, equity, and spending with state all-payer waiver priorities: A discrete choice experiment

OBJECTIVE

The state of Vermont has a statewide waiver from the centers for medicare and medicaid services to allow all-payer Accountable Care Organizations (ACOs). The Vermont all-payer model (VAPM) waiver is layered upon previous reforms establishing regional community health teams (CHTs) and medical homes. The waiver is intended to incentivize healthcare value and quality and create alignment between health system payers, providers, and CHTs. The objective of this study was to examine CHT's trade-offs and preferences for health, equity, and spending and the alignment with VAPM priorities.

DATA SOURCES/STUDY SETTING

Data were gathered from a survey and discrete choice experiment among CHT leadership and CHT team members of the 13 CHTs in Vermont.

STUDY DESIGN

We used conditional logit models to model the choice as a function of its characteristics (attributes) and mixed logit models to analyze whether preferences for programs varied by persons and roles within CHTs.

DATA COLLECTION/EXTRACTION METHODS

There were 60 respondents who completed the survey online with 14 choice tasks, with three program options in each task, for a total sample size of 2520.

PRINCIPAL FINDINGS

We found that CHTs prioritized programs in the community health plan and those with quantitative evidence of effectiveness. They were less likely to choose either programs targeting racial and ethnic minorities or programs having a small effect on a large population. Preferences did not vary across individual or community attributes. Program priorities of the VAPM, especially healthcare spending, were not prioritized.

CONCLUSIONS

The results suggest that the new VAPM does not automatically create system alignment: CHTs tended to prioritize local needs and voices. The statewide priorities are less important to CHTs, which have excellent internal alignment. This creates potential disconnection between state and community health goals. However, CHTs and the VAPM prioritize similar populations, indicating an opportunity to increase alignment by allowing flexible programs tailored to local needs. CHTs also prioritized programs with a strong evidence base, suggesting another potential avenue to create system alignment.

Estimating Surveillance Bias in Child Maltreatment Reporting During Home Visiting Program Involvement

It is unclear if surveillance bias (increased reports to Child Protective Services [CPS] related to program involvement) has a substantial impact on evaluation of home visiting (HV) prevention programs. We estimated surveillance bias using data from Connecticut's HV program, birth certificates, CPS, and hospitals. Using propensity score matching, we identified 15,870 families similar to 4015 HV families. The difference-in-differences approach was used to estimate surveillance bias as the change in investigated reports from the last 6 months of program involvement to the next 6 months. The median age of the children at program exit was 1.2 years (range: 60 days, 5 years). We estimated that 25.6% of investigated reports in the HV group resulted from surveillance bias. We reviewed CPS reports of 194 home-visited families to determine if a home visitor made the report and found that 10% were directly from home visitors. Program evaluations should account for surveillance bias.

Designing and Developing Online Training for Diabetes Prevention Program Coaches Using an Integrated Knowledge Translation Approach: Development and Usability Study

BACKGROUND

e-Learning has rapidly become a popular alternative to in-person learning due to its flexibility, convenience, and wide reach. Using a systematic and partnered process to transfer in-person training to an e-learning platform helps to ensure the training will be effective and acceptable to learners.

OBJECTIVE

This study aimed to develop an e-learning platform for Small Steps for Big Changes (SSBC) type 2 diabetes prevention program coaches to improve the viability of coach training.

METHODS

An integrated knowledge translation approach was used in the first 3 stages of the technology-enhanced learning (TEL) evaluation framework to address the study objective. This included three steps: (1) conducting a needs analysis based on focus groups with previously trained SSBC coaches, meetings with the SSBC research team, and a review of research results on the effectiveness of the previous in-person version of the training; (2) documenting processes and decisions in the design and development of the e-learning training platform; and (3) performing usability testing. Previously trained SSBC coaches and the SSBC research team were included in all stages of this study.

RESULTS

Step 1 identified components from the in-person training that should be maintained in the e-learning training (ie, a focus on motivational interviewing), additional components to be added to the e-learning training (ie, how to deliver culturally safe and inclusive care), and mode of delivery (videos and opportunities to synchronously practice skills). Step 2 documented the processes and decisions made in the design and development of the e-learning training, including the resources (ie, time and finances) used, the content of the training modules, and how coaches would flow through the training process. The design and development process consisted of creating a blueprint of the training. The training included 7 e-learning modules, the learning modalities of which included narrated demonstration videos and user-engaging activities, a mock session with feedback from the research team, and a final knowledge test. Step 3, usability testing, demonstrated high levels of learnability, efficiency, memorability, and satisfaction, with minor bugs documented and resolved.

CONCLUSIONS

Using an integrated knowledge translation approach to the technology-enhanced learning evaluation framework was successful in developing an e-learning training platform for SSBC coaches. Incorporating end users in this process can increase the chances that the e-learning training platform is usable, engaging, and acceptable. Future research will include examining the satisfaction of coaches using the SSBC coach e-learning training platform, assessing coach learning outcomes (ie, knowledge and behavior), and estimating the cost and viability of implementing this training.

Effectiveness of prevention programmes on the rate of burn injuries in children: a systematic review

INTRODUCTION

Burns are a frequent injury in children and can cause great physical and psychological impairment. Studies have identified positive effects of prevention measures based on increase in knowledge or reduction in hazards. The main goal of burn prevention campaigns, however, is to prevent burns. Therefore, this review is focused on the effectiveness of prevention programmes on the rates of burns in children.

METHODS

A literature search was performed on PubMed, Embase, CINAHL, Web of Science, Google Scholar and Scopus, including a reference-check. Included were studies which evaluated burn prevention programmes in terms of burn injury rate in children up to 19 years old. Studies specifically focused on non-accidental burns were excluded as well as studies with only outcomes such as safety knowledge or number of hazards.

RESULTS

The search led to 1783 articles that were screened on title and abstract. 85 articles were screened in full text, which led to 14 relevant studies. Nine of them reported a significant reduction in burn injury rate. Five others showed no effect on the number of burn injuries. In particular, studies that focused on high-risk populations and combined active with passive preventive strategies were successful.

CONCLUSION

Some prevention programmes appear to be an effective manner to reduce the number of burn injuries in children. However, it is essential to interpret the results of the included studies cautiously, as several forms of biases may have influenced the observed outcomes. The research and evidence on this subject is still very limited. Therefore, it is of great importance that future studies will be evaluated on a decrease in burns and bias will be prevented. Especially in low-income countries, where most of the burns in children occur and the need for effective prevention campaigns is vital.

The State of National Institute of Health Awards for funding genetic counseling research, resources, and training over the past decade

Research related to the practice of genetic counseling has historically been accomplished through student projects, small private grants, or as a part of a larger research study. Yet, recent initiatives supported by the National Society of Genetic Counselors and the National Human Genome Research Institute have recognized and promoted the need for additional genetic counseling research funding and training. In this study, we aimed to characterize awards from the United States' National Institutes of Health (NIH) over the past 10 years that support research related to genetic counseling. A search of the NIH RePORTER database conducted on April 25, 2022, using terms broadly related to genetic counseling, identified 3993 awards from fiscal years 2011 to 2022. After deduplication, 1231 grants were reviewed for screening. The final dataset included 43 awards of various types with the majority being research or R series grants [R01 (n = 17), R21 (n = 7)]. The remaining awards were Cooperative Agreements (U01, n = 6), intramural (Z-grants, n = 5), Center Core (P30, n = 2), Specialized Center (P50, n = 1), Career Development Award (K01, n = 1), Other Transactions (OT2, n = 1), Resource Program (G13, n = 1), and Research Training and Fellowship (T32, n = 1). Most grants were awarded between 2019 and 2022 (n = 24, 55.8%). The majority of awards were categorized as studies that included a comparison of different genetic counseling models (n = 23), outcomes of implementing genetic counseling (n = 11), core resources for genetic counselors (n = 5), and genetic counseling training programs (n = 4). Thirteen of the awards (30.2%) had a specific aim/goal/focus on underserved or underrepresented populations. The topics were predominantly related to cancer (n = 26). Nine awards were led or co-led by a genetic counselor (20.9%). Despite significant growth in genetic counseling research support from US-based funders over the past 3 years, major gaps related to funding exist, including that most award recipients are not genetic counselors.

Child-focused infant and early childhood mental health consultation: Shifting adult attributions to reduce the risk for preschool expulsion

Infant and early childhood mental health consultation (IECMHC) in early care and education (ECE) settings is a promising approach to support young children. Although research on the effects of IECMHC is encouraging, it is limited by the complexities of the systems in which IECMHC is implemented and the variability in IECMHC models. The current study aims to clearly articulate a statewide, child-focused, short-term IECMHC model, assess consultee satisfaction, examine the effects of consultation on children's functioning in the school and home settings, and evaluate changes in teacher perceptions associated with expulsion risk following consultation. In total, 268 children ages 1-5 (69% White, 75% male) and their family and school caregivers participated in consultation in a New England state, and 95 children and caregivers were included in an evaluation subsample. Of this subsample, teachers and ECE administrators, but not families, indicated significant improvement in children's functioning from referral to end of consultation. There was also a significant decrease in children's risk of expulsion, as measured by teachers' perceptions associated with expulsion decisions. This study contributes to the IECMHC literature by providing results specific to a child-focused model of consultation and highlighting the possible role of adult attributions for children in ECE.

Evaluation of the long-term outcomes of the University of Michigan's Practice-Oriented Research Training (PORT) program

The University of Michigan created the Practice-Oriented Research Training (PORT) program and implemented it between 2008 and 2018. The PORT program provided research training and funding opportunities for allied healthcare professionals. The program consisted of weekly didactics and group discussion related to topics relevant to developing specific research ideas into projects and funding for a mentored research project for those who submitted a competitive grant application. The goal of this evaluation was to assess the long-term impact of the PORT program on the research careers of the participants. Ninety-two participants (74 staff and 18 faculty) participated in both phases of the program. A mixed-methods approach to evaluation was used; 25 participants who received funding for their research completed surveys, and semi-structured interviews were conducted with eight program participants. In addition, data were collected on participants' publication history. Fifteen out of the 74 staff participants published 31 first-authored papers after participating in PORT. Twelve out of 15 staff participants who published first-authored papers did so for the first time after participating in the PORT program. Results of quantitative and qualitative analyses suggest that the PORT program had positive impacts on both participants and the research community.

Developing a Comprehensive List of Criteria to Evaluate the Characteristics and Quality of eHealth Smartphone Apps: Systematic Review

BACKGROUND

The field of eHealth is growing rapidly and chaotically. Health care professionals need guidance on reviewing and assessing health-related smartphone apps to propose appropriate ones to their patients. However, to date, no framework or evaluation tool fulfills this purpose.

OBJECTIVE

Before developing a tool to help health care professionals assess and recommend apps to their patients, we aimed to create an overview of published criteria to describe and evaluate health apps.

METHODS

We conducted a systematic review to identify existing criteria for eHealth smartphone app evaluation. Relevant databases and trial registers were queried for articles. Articles were included that (1) described tools, guidelines, dimensions, or criteria to evaluate apps, (2) were available in full text, and (3) were written in English, French, German, Italian, Portuguese, or Spanish. We proposed a conceptual framework for app evaluation based on the dimensions reported in the selected articles. This was revised iteratively in discussion rounds with international stakeholders. The conceptual framework was used to synthesize the reported evaluation criteria. The list of criteria was discussed and refined by the research team.

RESULTS

Screening of 1258 articles yielded 128 (10.17%) that met the inclusion criteria. Of these 128 articles, 30 (23.4%) reported the use of self-developed criteria and described their development processes incompletely. Although 43 evaluation instruments were used only once, 6 were used in multiple studies. Most articles (83/128, 64.8%) did not report following theoretical guidelines; those that did noted 37 theoretical frameworks. On the basis of the selected articles, we proposed a conceptual framework to explore 6 app evaluation dimensions: context, stakeholder involvement, features and requirements, development processes, implementation, and evaluation. After standardizing the definitions, we identified 205 distinct criteria. Through consensus, the research team relabeled 12 of these and added 11 more-mainly related to ethical, legal, and social aspects-resulting in 216 evaluation criteria. No criteria had to be moved between dimensions.

CONCLUSIONS

This study provides a comprehensive overview of criteria currently used in clinical practice to describe and evaluate apps. This is necessary as no reviewed criteria sets were inclusive, and none included consistent definitions and terminology. Although the resulting overview is impractical for use in clinical practice in its current form, it confirms the need to craft it into a purpose-built, theory-driven tool. Therefore, in a subsequent step, based on our current criteria set, we plan to construct an app evaluation tool with 2 parts: a short section (including 1-3 questions/dimension) to quickly disqualify clearly unsuitable apps and a longer one to investigate more likely candidates in closer detail. We will use a Delphi consensus-building process and develop a user manual to prepare for this undertaking.

TRIAL REGISTRATION

PROSPERO International Prospective Register of Systematic Reviews CRD42021227064; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021227064.

Experiences and perspectives of integrating nutrition education into an exercise program for people with chronic medical conditions

ISSUE ADDRESSED

Community-based healthy eating and exercise programs are effective interventions to support some people with chronic conditions. This research aimed to explore the experiences and perceptions of participants regarding the integration of nutrition education into a successful group based exercise program for people with chronic medical conditions.

METHODS

Semi-structured interviews were conducted with past participants of a community based group exercise program (Health Moves) with embedded nutrition advice based in a regional area of New South Wales. A total of 60 individuals who had participated in the Health Moves program between 2017 and 2019 were invited to participate and 19 (32%) randomly selected participants consented to be interviewed. Interviews explored participants' experiences, program impacts, and barriers to sustaining changes post program. Similar concepts and patterns were grouped into themes.

RESULTS

Four themes were identified that described the experiences with a community-based group exercise intervention with a nutrition component. The major themes evident were that Health Moves facilitated motivation (via access to health professionals, peer support, accountability, affordability); and there were challenges identified to sustaining change (including cost, comorbidities, end of program support, environmental factors); Nutrition advice was valuable for some but not for all (difficulty recalling nutrition components, superficial coverage) Practical and interactive nutrition advice is desired.

CONCLUSIONS

Participants expressed a high degree of satisfaction with the program but require ongoing support to sustain changes post program. The position and integration of nutrition education within this exercise program was perceived by participants to be suboptimal. Increased access to practical, interactive nutrition education components may improve participant satisfaction and engagement. SO WHAT?: Key findings from this research include a desire for removal of didactic nutrition education sessions and request for increased peer support. Modifications to the program include the integration of interactive self-paced nutrition modules. Peer support partnerships are now encouraged by trainers to support ongoing motivation of participants to keep training together outside the structured exercise program and transition to managing their own exercise routine. Discussions between organisations involved about ongoing pathway/program support or reduced cost 'off peak' gym membership is underway to help with costs incurred by participants.

Generating Contextual Variables From Web-Based Data for Health Research: Tutorial on Web Scraping, Text Mining, and Spatial Overlay Analysis

BACKGROUND

Contextual variables that capture the characteristics of delimited geographic or jurisdictional areas are vital for health and social research. However, obtaining data sets with contextual-level data can be challenging in the absence of monitoring systems or public census data.

OBJECTIVE

We describe and implement an 8-step method that combines web scraping, text mining, and spatial overlay analysis (WeTMS) to transform extensive text data from government websites into analyzable data sets containing contextual data for jurisdictional areas.

METHODS

This tutorial describes the method and provides resources for its application by health and social researchers. We used this method to create data sets of health assets aimed at enhancing older adults' social connections (eg, activities and resources such as walking groups and senior clubs) across the 374 health jurisdictions in Catalonia from 2015 to 2022. These assets are registered on a web-based government platform by local stakeholders from various health and nonhealth organizations as part of a national public health program. Steps 1 to 3 involved defining the variables of interest, identifying data sources, and using Python to extract information from 50,000 websites linked to the platform. Steps 4 to 6 comprised preprocessing the scraped text, defining new variables to classify health assets based on social connection constructs, analyzing word frequencies in titles and descriptions of the assets, creating topic-specific dictionaries, implementing a rule-based classifier in R, and verifying the results. Steps 7 and 8 integrate the spatial overlay analysis to determine the geographic location of each asset. We conducted a descriptive analysis of the data sets to report the characteristics of the assets identified and the patterns of asset registrations across areas.

RESULTS

We identified and extracted data from 17,305 websites describing health assets. The titles and descriptions of the activities and resources contained 12,560 and 7301 unique words, respectively. After applying our classifier and spatial analysis algorithm, we generated 2 data sets containing 9546 health assets (5022 activities and 4524 resources) with the potential to enhance social connections among older adults. Stakeholders from 318 health jurisdictions registered identified assets on the platform between July 2015 and December 2022. The agreement rate between the classification algorithm and verified data sets ranged from 62.02% to 99.47% across variables. Leisure and skill development activities were the most prevalent (1844/5022, 36.72%). Leisure and cultural associations, such as social clubs for older adults, were the most common resources (878/4524, 19.41%). Health asset registration varied across areas, ranging between 0 and 263 activities and 0 and 265 resources.

CONCLUSIONS

The sequential use of WeTMS offers a robust method for generating data sets containing contextual-level variables from internet text data. This study can guide health and social researchers in efficiently generating ready-to-analyze data sets containing contextual variables.

Assessing Nurse Competency in the Veterans Health Administration Registered Nurse Transition-to-Practice Residency Program: Item Pool Content Validation

BACKGROUND

New graduate registered nurse (RN) competencies are complex and difficult to measure. Psychometrically sound tools are needed to evaluate competencies for nurses participating in nurse residencies.

PURPOSE

Project aims were to develop an item pool for a tool to measure new graduate RN competencies for the Veterans Health Administration RN Transition-to-Practice Residency Program; validate item pool content; and use consensus methods to improve item pool content validity.

METHODS

A sequential, mixed-methods design was used. Item pool creation, content validation, and revisions included a collaborative process with the evaluation team, operational stakeholders, and subject matter experts (SMEs).

RESULTS

Inclusion of SMEs in item development enhanced item pool content validity to measure nurse competency. Stakeholder feedback ensured programmatic logistical and evaluation concerns were met.

CONCLUSIONS

Engaging SMEs in conceptualization, item development, and aligning existing standards enhanced item pool content validity to measure nurse competencies for new graduate RNs.

Determinants of Bluetooth-Enabled Self-Measured Blood Pressure Monitoring in Federally Qualified Health Centers

BACKGROUND

In 2021, the Health Resources and Services Administration (HRSA) launched the National Hypertension Control Initiative (HTN Initiative) with the goal to enhance HTN control through Bluetooth-enabled self-measured blood pressure (BT-SMBP) monitoring and use this data to inform clinical decisions in Federally Qualified Health Centers (FQHCs) with a large proportion of their population with uncontrolled blood pressure (BP). We sought to understand the experience of Michigan-based FQHCs in implementing the HTN initiative.

METHODS

Staff from three Michigan-based FQHCs were invited to participate in semi-structured interviews from September to November 2022. Interviews were conducted in-person and were based on the Tailored Implementation in Chronic Diseases framework. Content analysis was performed by three coders.

RESULTS

Ten staff participated in interviews (FQHC 1: n = 6, FQHC 2: n = 1, FQHC 3: n = 3). The FQHCs differed in their stage of implementation and their approach. FQHC 1 created a large-scale, community health worker driven program, FQHC 2 created a small-scale, short term, BP device loan program, and FQHC 3 created a primarily outsourced, large-scale program through a contracted partner. Positive staff attitudes and outcome expectations, previous experience with SMBP grants, supportive clinic leadership, social support, and free BP cuff resources were identified as facilitators to implementation. Patients' high social needs, SMBP-related Technology, and insufficient workforce and staff capacity were identified as barriers.

CONCLUSION

BT-SMBP among FQHC patients is promising but challenges in integrating SMBP data into clinic workflow, workforce capacity to support the high social needs of participants, and to assist in reacting to the more frequent BP data remain to be overcome.

Professionalism Program Evaluation and Its Impact on Undergraduate Medical Students

Objective The study aims to evaluate the effectiveness of a professionalism program by conducting focus group discussions (FGDs). The research focuses on understanding how the program influences the development of professionalism skills among medical students, as well as their perceptions and experiences regarding the program. The study's objectives revolve around identifying the strengths and weaknesses of the program from the perspective of the students. Methods This was a qualitative study done after obtaining approval from the Institutional Review Committee, Islamic International Medical College. Students of 4th year Bachelor of Medicine and Bachelor of Surgery (MBBS; 56 students) were selected for the study. A convenience sampling technique was used to select the participants for FGDs. Seven separate FGDs were conducted, with eight participants in each group. All the FGDs were audiotaped. Data were transcribed and translated. Data were analyzed using the thematic analysis on Atlas. Both obvious and hidden content were analyzed. Codes and themes were identified, which were then finalized with consensus. Codes were then categorized into sub-themes, and finally, themes were generated. Results Participants identified the problems associated with applying professionalism during FGDs. These challenges encompassed hefty workloads and a perceived disparity between theoretical knowledge and practical implementation. The students highlighted the importance of practical teaching methods, the cultivation of role models, the alignment of curriculum with real-world experiences, and the revision of assessment methods. This study analyzes the obstacles and potential advantages associated with professionalism education and presents significant perspectives on improving it for upcoming healthcare practitioners. Conclusion Professionalism is a crucial component, and each graduate of a medical school should be fully qualified as a professional after graduation. As we want our future doctors to be skilled at both professional qualities and diagnosis, it is crucial that medical institutions teach professionalism.

Implementing and Enhancing Social and Economic Needs Screening at a Federally Qualified Health Center

BACKGROUND

Programs to screen for social and economic needs (SENs) are challenging to implement.

AIM

To describe implementation of an SEN screening program for patients obtaining care at a federally qualified health center (FQHC).

SETTING

Large Chicago-area FQHC where many patients are Hispanic/Latino and insured through Medicaid.

PROGRAM DESCRIPTION

In the program's phase 1 (beginning April 2020), a prescreening question asked about patients' interest in receiving community resources; staff then called interested patients. After several refinements (e.g., increased staffing, tailored reductions in screening frequency) to address challenges such as a large screening backlog, program phase 2 began in February 2021. In phase 2, a second prescreening question asked about patients' preferred modality to learn about community resources (text/email versus phone calls).

PROGRAM EVALUATION

During phase 1, 8925 of 29,861 patients (30%) expressed interest in community resources. Only 40% of interested patients were successfully contacted and screened. In phase 2, 5781 of 21,737 patients (27%) expressed interest in resources; 84% of interested patients were successfully contacted by either text/email (43%) or phone (41%).

DISCUSSION

Under one-third of patients obtaining care at an FQHC expressed interest in community resources for SENs. After program refinements, rates of follow-up with interested patients substantially increased.

Engaging health plans to prioritize HPV vaccination and initiate at age 9

Health plans can influence pediatric and primary care providers and patients to understand HPV vaccination coverage and increase HPV vaccination uptake. By initiating vaccination at age nine, health plans can lay the groundwork for on-time HPV cancer prevention by age 13. In 2022, the American Cancer Society engaged 28 health plans in a 12-month HPV vaccination learning collaborative in which plans set their own quality improvement targets, implemented multi-pronged interventions, and joined quarterly best-practice sharing calls. Twenty-five of the 28 plans reported including a focus on ages 9 to 10. Preliminary pre-intervention data illustrate that vaccination rates from participating plans follow national trends and reaffirm existing gaps for HPV vaccination. Health plan interventions to address HPV vaccination are consistent with best practices but could be maximized to target initiation at ages 9-10 by using provider and patient reminders, targeted provider education, and dose-specific provider pay for performance and patient incentive programs. Health plans should explore future capacity to analyze non-HEDIS required data, including HPV initiation and HPV vaccination data for adolescents below age 13.

Enhancing Workforce Pathways: Insights From a Paid Internship Pilot Program Linking Public Health Students and Local Health Departments

The persistent understaffing of the governmental public health workforce has led to program cutbacks, staff burnout at local health departments (LHDs), and an urgent need to replenish staffing. To build recruitment pathways into LHDs and build their workforce capacity, we introduced a paid internship initiative connecting Master's in Public Health students from a Midwestern university with LHDs in the state. This article presents the pilot program developed and the insights gained from it. Program participants included nine LHDs that hosted 10 students for 12-week internships. Internship projects were developed by LHDs with support from the state's association of county and city health officials. All students completed their internship projects satisfactorily. The experience highlighted that while students contributed to LHDs through short-term projects, with sustained backing and minor adjustments, this model can serve to reinforce the governmental public health sector's existing and future capacity in the long term.

Updating and evaluating a research best practices training course for social and behavioral research professionals

Introduction

The clinical and translational research workforce involved in social and behavioral research (SBR) needs to keep pace with clinical research guidance and regulations. Updated information and a new module on community and stakeholder engagement were added to an existing SBR training course. This article presents evaluation findings of the updated course for the Social and Behavioral Workforce.

Methods and Materials

Participants working across one university were recruited. Course completers were sent an online survey to evaluate the training. Some participants were invited to join in a focus group to discuss the application of the training to their work. We performed descriptive statistics and conducted a qualitative analysis on focus group data.

Results

There were 99 participants from diverse backgrounds who completed the survey. Most reported the training was relevant to their work or that of the study teams they worked with. Almost half (46%) indicated they would work differently after participating. Respondents with community or stakeholder engaged research experience vs. those without were more likely to report that the new module was relevant to study teams they worked with (t = 5.61, p = 0.001), and that they would work differently following the training (t = 2.63, p = 0.01). Open-ended survey responses (n = 99) and focus group (n = 12) data showed how participants felt their work would be affected by the training.

Conclusion

The updated course was rated highly, particularly by those whose work was related to the new course content. This course provides an up-to-date resource for the training and development for the Social and Behavioral Workforce.

Say Something, Do Something: Evaluating a Forum Theater Production to Activate Youth Violence Prevention Strategies in Schools

BACKGROUND

Youth violence that takes place within school settings exposes youth to serious social, mental and physical consequences that affect education performance, and life opportunities. Previous work shows positive youth development frameworks can promote social-emotional learning by enhancing empathy and building problem-solving and conflict management skills. Theater-based interventions have been shown to enhance social emotional development by privileging youth voices, and building youth capacities and strengths. The current manuscript presents the evaluation of an arts-based and public health framework conducted to assess the development, implementation and impact of a forum theater production, Say Something, Do Something (SSDS) in St. Louis, Missouri.

METHODS

An iterative mixed methods approach was used, starting with observations of productions. Using convenience sampling, we then conducted post interviews of the theater team (n = 8) and school personnel (n = 10).

RESULTS

Respondents highlighted that as a result of engagement of school personnel in program development, the language and scenarios presented were relevant to students. Data indicated that SSDS increased student knowledge and changed attitudes, developed student conflict management and problem-solving skills, and improved interpersonal behavior. SSDS also raised awareness of the importance of, and created the foundation for, additional system and policy changes in the schools.

CONCLUSION AND IMPLICATIONS

Forum theater is an approach that can enhance socio-emotional learning and conflict management among youth. Collaborative initiatives between public health and the arts are poised to uniquely engage community partners, animate interventions, and impact critical public health issues including youth violence prevention.

The value and impact of health technology assessment: discussions and recommendations from the 2023 Health Technology Assessment International Global Policy Forum

Health technology assessment (HTA) programs inform decision making about the value and reimbursement of new and existing health technologies; however, they are under increasing pressure to demonstrate that they are a cost-effective use of finite healthcare resources themselves. The 2023 HTAi Global Policy Forum (GPF) discussed the value and impact of HTA, including how it is assessed and communicated, and how it could be enhanced in the future. This article summarizes the discussions held at the 2023 HTAi GPF, where the challenges and opportunities related to the value and impact of HTA were debated. Core themes and recommendations identified that defining the purpose of value and impact assessment is an essential first step prior to undertaking it, and that it can be done through the use and expansion of existing tools. Further work around aligning HTA programs with underlying societal values is needed to ensure the long-term value and impact of HTA. HTA could also have a role in assessing the efficiency of the wider health system by applying HTA methods or concepts to broader budgetary allocations and organizational aspects of health care. Stakeholders (particularly patients, industry, and clinicians but also payers, wider society, and the media) should ideally be actively engaged when undertaking the value and impact assessment of HTA. More concerted efforts in communicating the role and remit of HTA bodies would also help stakeholders to better understand the value and impact of HTA, which in turn could improve the implementation of HTA recommendations and application to future actions in the lifecycle of technologies.

Participation in a Medicare advanced primary care model and the delivery of high-value services

OBJECTIVE

To evaluate whether primary care providers' participation in the Comprehensive Primary Care Plus Initiative (CPC+) was associated with changes in their delivery of high-value services.

DATA SOURCES

Medicare Physician & Other Practitioners public use files from 2013 to 2019, 2017 to 2019 Medicare Part B claims for a 5% random sample of Medicare Fee-for-Service (FFS) beneficiaries, the Area Health Resources File, the National Plan & Provider Enumeration System files, and public use datasets from the Centers for Medicare & Medicaid Services Physician Compare.

STUDY DESIGN

We used a difference-in-difference approach with a propensity score-matched comparison group to estimate the association of CPC+ participation with the delivery of annual wellness visits (AWVs), advance care planning (ACP), flu shots, counseling to prevent tobacco use, and depression screening. These services are prominent examples of high-value services, providing benefits to patients at a reasonable cost. We examined both the likelihood of delivering these services within a year and the count of services delivered per 1000 Medicare FFS beneficiaries per year.

DATA COLLECTION/EXTRACTION METHODS

Secondary data are linked at the provider level.

PRINCIPAL FINDINGS

We find that CPC+ participation was associated with increases in the likelihood of delivering AWVs (13.03 percentage points by CPC+'s third year, p < 0.001) and the number of AWVs per 1000 Medicare FFS beneficiaries (44 more AWVs by CPC+'s third year, p < 0.001). We also find that CPC+ participation was associated with more flu shots per 1000 beneficiaries (52 more shots by CPC+'s third year, p < 0.001) but not with the likelihood of delivering flu shots. We did not find consistent evidence for the association between CPC+ participation and ACP services, counseling to prevent tobacco use, or depression screening.

CONCLUSIONS

CPC+ participation was associated with increases in the delivery of AWVs and flu shots, but not other high-value services.

Incorporating telehealth education into the genetic counseling curriculum

As the provision of telehealth genetic counseling (THGC) services continues to expand, it is imperative that genetic counseling students gain proficiency in telehealth service delivery. To prepare students to provide THGC services, the MSGC program at the University of Pennsylvania has included didactic sessions on THGC, THGC role plays, THGC standardized patient sessions, and THGC fieldwork experiences and clinical rotations. This article highlights best practices in THGC and guidance for Master of Science in Genetic Counseling (MSGC) programs training the next generation of genetic counselors providing THGC services.

Network Analysis for Formative Evaluation of Collaborative, Team Science Research Partnerships

Cancer health disparities persist across the cancer care continuum despite decades of effort to eliminate them. Among the strategies currently used to address these disparities are multi-institution research initiatives that engage multiple stakeholders and change efforts. Endemic to the theory of change of such programs is the idea that collaboration-across institutions, research disciplines, and academic ranks-is necessary to improve outcomes. Despite this emphasis on collaboration, however, it is not often a focus of evaluation for these programs and others like them. In this paper we describe a method for evaluating collaboration within the Meharry-Vanderbilt-Tennessee State University Cancer Partnership using network analysis. Specifically, we used network analysis of co-authorship on academic publications to visualize the growth and patterns of scientific collaboration across partnership institutions, research disciplines, and academic ranks over time. We presented the results of the network analysis to internal and external advisory groups, creating the opportunity to discuss partnership collaboration, celebrate successes, and identify opportunities for improvement. We propose that basic network analysis of existing data along with network visualizations can foster conversation and feedback and are simple and effective ways to evaluate collaboration initiatives.