Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing Examination

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Introduction

The accurate assessment of health professionals' competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates' competence of health professionals.

Methods

The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data.

Results

The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson's correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850).

Discussion

In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates' competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates' competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.

The accuracy and consistency of mastery for each content domain using the Rasch and deterministic inputs, noisy “and” gate diagnostic classification models: a simulation study and a real-world analysis using data from the Korean Medical Licensing Examination

PURPOSE

Diagnostic classification models (DCMs) were developed to identify the mastery or non-mastery of the attributes required for solving test items, but their application has been limited to very low-level attributes, and the accuracy and consistency of high-level attributes using DCMs have rarely been reported compared with classical test theory (CTT) and item response theory models. This paper compared the accuracy of high-level attribute mastery between deterministic inputs, noisy “and” gate (DINA) and Rasch models, along with sub-scores based on CTT.

METHODS

First, a simulation study explored the effects of attribute length (number of items per attribute) and the correlations among attributes with respect to the accuracy of mastery. Second, a real-data study examined model and item fit and investigated the consistency of mastery for each attribute among the 3 models using the 2017 Korean Medical Licensing Examination with 360 items.

RESULTS

Accuracy of mastery increased with a higher number of items measuring each attribute across all conditions. The DINA model was more accurate than the CTT and Rasch models for attributes with high correlations (>0.5) and few items. In the real-data analysis, the DINA and Rasch models generally showed better item fits and appropriate model fit. The consistency of mastery between the Rasch and DINA models ranged from 0.541 to 0.633 and the correlations of person attribute scores between the Rasch and DINA models ranged from 0.579 to 0.786.

CONCLUSION

Although all 3 models provide a mastery decision for each examinee, the individual mastery profile using the DINA model provides more accurate decisions for attributes with high correlations than the CTT and Rasch models. The DINA model can also be directly applied to tests with complex structures, unlike the CTT and Rasch models, and it provides different diagnostic information from the CTT and Rasch models.

A Seed Usage Issue on Using catR for Simulation and the Solution

The R package catR is a useful developmental and testing platform for adaptive tests. Developers and researchers have been using catR for their assessments or research projects. However, there is a flaw in catR that can potentially cause misleading results. This article shows the flaw and provides a solution. Suggestions in using seeds for random numbers were also provided.

Liner programming method to construct equated item sets for implementation of periodical computer-based testing of Korean medical licensing examination

PURPOSE

This study aimed to find the best way of developing equivalent item sets and to propose a stable and effective management plan for the periodical licensing examinations.

METHODS

Five pre-equated item sets were developed based on the predicted correct answer rate of each item by using linear programming. These pre-equated item sets were compared to the ones that were developed with random item selection method based on the actual answer rate and difficulty from item response theory (IRT). Also, the results with and without common items were compared in the same way. ACAR and the IRT difficulty was used to determine whether there is a significant difference between pre-equating conditions.

RESULTS

There was a statistically significant difference in IRT difficulty among the results from different pre-equated conditions. As predicted correct answer rate was divided into 2 or 3 difficulty boundaries, the actual answer rate and IRT difficulty parameters of the 5 item sets were equally constructed. Comparing item sets conditions with common items and without common items, including common items did not contribute much for the equating of 5 item sets.

CONCLUSION

This study suggested the linear programming method is applicable to construct equated-item sets that reflect each content area. The best method to construct equated item sets suggested is to divide the predicted correct answer rate into 2 or 3 difficulty boundaries regardless of common items. If pre-equated item sets are required to construct a test based on the actual data, several optimal methods should be considered by simulation studies before administrating a real test.