Designing and evaluating complex educational research to meet federal grant requirement: A demonstration of a funded large-scale protocol

Federal funding agencies have invested significant resources supporting evidence-based, innovative approaches that address education problems and incorporate rigorous design and evaluation, particularly through randomized controlled trials (RCTs), the gold standard for yielding causal inference in scientific research. In this study, we introduced factors (i.e., evaluation design, attrition, outcome measures, analytic approach, and fidelity of implementation) that are often times required in the Federal Notice for application by the U.S. Department of Education, with an emphasis on What Works Clearinghouse (WWC) standards. We further presented a federally funded research protocol with a multi-year, clustered RCT design to determine the impact of an instructional intervention on students' academic performance in high-needs schools. In the protocol, we elaborated on how our research design, evaluation plan, power analysis, as well as confirmatory research questions and analytical approaches were aligned with the grant requirement and WWC standards. We intend to provide a road map to meeting WWC standards and to increase the likelihood of successful grant applications.

Identifying and Reducing Disparities in Mental Health Outcomes Among American Indians and Alaskan Natives Using Public Health, Mental Healthcare and Legal Perspectives

The purpose of this paper was to investigate disparities in mental healthcare delivery in American Indian/Alaska Native populations from three perspectives: public health, legal policy and mental healthcare and provide evidence-based recommendations toward reducing those disparities. Data on mental health funding to tribes were obtained from the Substance Abuse and Mental Health Services Administration. As a result of analysis of these data, vital statistics and current literature, we propose three recommendations to reduce mental health disparities. First, where possible, increase mental health funding opportunities for federally-recognized tribes. Second, model funding practices on principles of tribal self-determination. Finally, support diverse interventions that are culturally-based and culturally-appropriate.

The opportunities and challenges of multi-site evaluations: lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery-Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.

From mission to measures: performance measure development for a Teen Pregnancy Prevention Program

The Office of Adolescent Health (OAH) sought to create a comprehensive set of performance measures to capture the performance of the Teen Pregnancy Prevention (TPP) program. This performance measurement system needed to provide measures that could be used internally (by both OAH and the TPP grantees) for management and program improvement as well as externally to communicate the program's progress to other interested stakeholders and Congress. This article describes the selected measures and outlines the considerations behind the TPP measurement development process. Issues faced, challenges encountered, and lessons learned have broad applicability for other federal agencies and, specifically, for TPP programs interested in assessing their own performance and progress.