Issues of rigor and feasibility when observing the quality of program implementation: a case study

Program evaluators have paid little attention in the literature to the manner in which measuring the quality of implementation with observations requires tradeoffs between rigor (reliability and validity) and program evaluation feasibility. We present a case example of how we addressed rigor in light of feasibility concerns when developing and conducting observations for measuring the quality of implementation of a small education professional development program. We discuss the results of meta-evaluative analyses of the reliability of the quality observations, and we present conclusions about conducting observations in a rigorous and feasible manner. The results show that the feasibility constraints that we faced did not notably reduce the rigor of our methods.

The development and validation of The Inquiry Science Observation Coding Sheet

Evaluation reports increasingly document the degree of program implementation, particularly the extent to which programs adhere to prescribed steps and procedures. Many reports are cursory, however, and few, if any, fully portray the long and winding path taken when developing evaluation instruments, particularly observation instruments. In this article, we describe the development of an observational method for evaluating the degree to which K-12 inquiry science programs are implemented, including the many steps and decisions that occurred during the development, and present evidence for the reliability and validity of the data that we collected with the instrument. The article introduces a method for measuring the adherence of inquiry science implementation and gives evaluators a full picture of what they might expect when developing observation instruments for assessing the degree of program implementation.

Involving program stakeholders in reviews of evaluators' recommendations for program revisions

Why is the use of stakeholder expertise a sound rationale for involving program staff and beneficiaries in reviewing evaluators' recommendations for program revisions? What procedures might evaluators use for involving these stakeholders in this task? What can we conclude about the likely success of these procedures? This article seeks to address these questions. In the context of educational program evaluation, some of the theoretical and practical issues about the participation of stakeholders in reviewing recommendations are discussed. The participation of program stakeholders for the purpose of tapping their expertise is explained; a recent evaluation, in which program stakeholders reviewed evaluators' recommendations for program revisions, is described; and the implications of the article for stakeholder-based evaluation and practical participatory evaluation are presented.