Improving progress test score estimation using bayesian statistics

Written-Based Progress Testing: A Scoping Review

PURPOSE

Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education.

METHOD

The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process.

RESULTS

A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge).

CONCLUSIONS

Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use.

Red, yellow and green: What does it mean? How the progress test informs and supports student progress

OBJECTIVES

Most medical schools using progress tests (PTs) provide feedback by utilizing a traffic-light system of green (satisfactory), yellow (borderline) and red (unsatisfactory) categories. There is little research assessing students' perceptions or usage of this feedback. Therefore this study proposed to determine the effectiveness of formative PTs at informing and supporting student progress.

METHODS

A mixed methods study was performed, involving a retrospective analysis of a results database to establish the predictive validity of PT categories and 11 semi-structured interviews to explore students' perceptions of PT feedback in a graduate entry medical programme.

RESULTS

Quantitative analysis revealed that students who always scored green performed better in their summative exams and graduated with a higher final degree than those who received a yellow or red category at least once. Qualitative analysis revealed that just over half of the interviewed students perceived the PT as having informed their progress. Most participants agreed that the current feedback is insufficient and doesn't guide their on-going learning.

CONCLUSION

While this study demonstrated that the PT is a useful predictive tool for informing student progress, in its current format it's not fulfilling a truly formative role and supporting student progress sufficiently.

Predicting neonatal intubation competency in trainees

BACKGROUND AND OBJECTIVES

Pediatric residency training programs are graduating residents who are not competent in neonatal intubation, a vital skill needed for any pediatrician involved in delivery room resuscitations. However, a precise definition of competency during training is lacking. The objective of this study was to more precisely define the trajectory toward competency in neonatal intubation for pediatric residents, as a framework for later evaluating complementary training tools.

METHODS

This is a retrospective single-center observational study of resident-performed neonatal intubations at Duke University Medical Center between 2005 and 2013. Using a Bayesian statistical model, intubation competency was defined when the resident attained a 75% likelihood of intubating their next patient successfully.

RESULTS

A total of 477 unique intubation attempts by 105 residents were analyzed. The path to proficiency was defined by a categorical or milestone learning event after which all learners move toward competency in a similar manner. In our cohort, 4 cumulative successes were needed to achieve competency. Only 24 of 105 (23%) achieved competency during the study period. Residents who failed their first 2 opportunities, compared with those successful on their first 2 opportunities, needed nearly double the intubation exposure to achieve competency.

CONCLUSIONS

Bayesian statistics may be useful to more precisely describe neonatal intubation competency in residents. Achieving competency in neonatal intubation appears to be a categorical or milestone learning event whose timing varies between residents. The current educational environment does not provide adequate procedural exposure to achieve competency for most residents.

The use of progress testing

Progress testing is gaining ground rapidly after having been used almost exclusively in Maastricht and Kansas City. This increased popularity is understandable considering the intuitive appeal longitudinal testing has as a way to predict future competence and performance. Yet there are also important practicalities. Progress testing is longitudinal assessment in that it is based on subsequent equivalent, yet different, tests. The results of these are combined to determine the growth of functional medical knowledge for each student, enabling more reliable and valid decision making about promotion to a next study phase. The longitudinal integrated assessment approach has a demonstrable positive effect on student learning behaviour by discouraging binge learning. Furthermore, it leads to more reliable decisions as well as good predictive validity for future competence or retention of knowledge. Also, because of its integration and independence of local curricula, it can be used in a multi-centre collaborative production and administration framework, reducing costs, increasing efficiency and allowing for constant benchmarking. Practicalities include the relative unfamiliarity of faculty with the concept, the fact that remediation for students with a series of poor results is time consuming, the need to embed the instrument carefully into the existing assessment programme and the importance of equating subsequent tests to minimize test-to-test variability in difficulty. Where it has been implemented-collaboratively-progress testing has led to satisfaction, provided the practicalities are heeded well.

A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71

There has been increasing use and significance of progress testing in medical education. It is used in many ways and with several formats to reflect the variety of curricula and assessment purposes. These developments have occurred alongside a recognised sensitivity for error variance inherent in multiple choice tests from which challenges to its validity and reliability have arisen. This Guide presents a generic, systemic framework to help identify and explore improvements in the quality and defensibility of progress test data. The framework draws on the combined experience of the Dutch consortium, an individual medical school in the United Kingdom, and the bulk of the progress test literature to date. It embeds progress testing as a quality-controlled assessment tool for improving learning, teaching and the demonstration of educational standards. The paper describes strengths, highlights constraints and explores issues for improvement. These may assist in the establishment of potential or new progress testing in medical education programmes. They can also guide the evaluation and improvement of existing programmes.