Detecting cheating in written medical examinations by statistical analysis of similarity of answers: pilot study

Item pre-knowledge true prevalence in clinical anatomy - application of gated item response theory model

BACKGROUND

Computer and paper examinations in our days are constructed from an item pool which is regularly updated. Given the way that exams are created, one of the major concerns is the security of the items that are being used in order to ensure a good estimation of abilities. The aim of this study is to measure the prevalence of item pre-knowledge in our medical school.

METHODS

The Deterministic, Gated Item Response Theory Model (DGM) was applied to estimate the prevalence of students who have had item pre-knowledge from six multiple choice examinations of the Clinical Anatomy course at the Faculty of Medicine of University of Porto. Each examination consisted of 100 items with an average of 200 students and 20% repeated items per examination. The estimation of the sensitivity and specificity was based on a simulation study. The sensitivity and specificity estimates, and apparent prevalence were used to estimate true prevalence of cheating students in the examinations under study.

RESULTS

The specificity in the DGM for different simulation scenarios was between 68 and 98%, while the sensitivity ranged from 60 to 91%. The apparent prevalence was between 0.0 and 3.4%, while the true prevalence ranged from 1.2 to 3.7%.

CONCLUSIONS

The true prevalence was much lower compared to the students self-reported copying of responses from other students; however, it is important to keep monitoring the pre-knowledge prevalence in order to enforce measures in case an increase occurs.

Investigating the existence of social networks in cheating behaviors in medical students

BACKGROUND

Most studies on academic cheating rely on self-reported questionnaires and focus on the individual, overlooking cheating as a group activity. The aim of this study is to estimate the true prevalence of cheating/anomalies among medical students using a statistical index developed for this purpose, and to explore the existence of social networks between anomalies in students' results.

METHODS

Angoff's A index was applied to a sample of 30 written examinations, with a total of 1487 students and 7403 examinations taken, from the 2014/2015 academic year of the Faculty of Medicine of the University of Porto to detect anomaly pairs. All analyses are within the same academic year and not across years. Through simulations, the sensitivity and specificity of the statistical method was determined, and the true prevalence of anomalies/cheating was estimated. Networks of anomaly pairs were created to search for patterns and to calculate their density.

RESULTS

The percentage of students who cheated at least once increased with the year of medical school, being lowest in the first year (3.4%) and highest in the fifth (17.3%). The year of medical school was associated with anomalies (p < 0.05). The network's density was also lowest in the first year (1.12E-04) and highest in the fifth (8.20E-04). The true prevalence of anomalies was estimated to be 1.85% (95%CI: 1.07-3.20%).

CONCLUSIONS

These findings suggest that some students are involved in social networks of cheating, which grow over time, resulting in an increase of anomalies/cheating in later academic years.

Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations

BACKGROUND

Bias of clinical examiners against some types of candidate, based on characteristics such as sex or ethnicity, would represent a threat to the validity of an examination, since sex or ethnicity are 'construct-irrelevant' characteristics. In this paper we report a novel method for assessing sex and ethnic bias in over 2000 examiners who had taken part in the PACES and nPACES (new PACES) examinations of the MRCP(UK).

METHOD

PACES and nPACES are clinical skills examinations that have two examiners at each station who mark candidates independently. Differences between examiners cannot be due to differences in performance of a candidate because that is the same for the two examiners, and hence may result from bias or unreliability on the part of the examiners. By comparing each examiner against a 'basket' of all of their co-examiners, it is possible to identify examiners whose behaviour is anomalous. The method assessed hawkishness-doveishness, sex bias, ethnic bias and, as a control condition to assess the statistical method, 'even-number bias' (i.e. treating candidates with odd and even exam numbers differently). Significance levels were Bonferroni corrected because of the large number of examiners being considered.

RESULTS

The results of 26 diets of PACES and six diets of nPACES were examined statistically to assess the extent of hawkishness, as well as sex bias and ethnicity bias in individual examiners. The control (odd-number) condition suggested that about 5% of examiners were significant at an (uncorrected) 5% level, and that the method therefore worked as expected. As in a previous study (BMC Medical Education, 2006, 6:42), some examiners were hawkish or doveish relative to their peers. No examiners showed significant sex bias, and only a single examiner showed evidence consistent with ethnic bias. A re-analysis of the data considering only one examiner per station, as would be the case for many clinical examinations, showed that analysis with a single examiner runs a serious risk of false positive identifications probably due to differences in case-mix and content-specificity.

CONCLUSIONS

In examinations where there are two independent examiners at a station, our method can assess the extent of bias against candidates with particular characteristics. The method would be far less sensitive in examinations with only a single examiner per station as examiner variance would be confounded with candidate performance variance. The method however works well when there is more than one examiner at a station and in the case of the current MRCP(UK) clinical examination, nPACES, found possible sex bias in no examiners and possible ethnic bias in only one.

Integrating personalized medical test contents with XML and XSL-FO

BACKGROUND

In 2004 the adoption of a modular curriculum at the medical faculty in Muenster led to the introduction of centralized examinations based on multiple-choice questions (MCQs). We report on how organizational challenges of realizing faculty-wide personalized tests were addressed by implementation of a specialized software module to automatically generate test sheets from individual test registrations and MCQ contents.

METHODS

Key steps of the presented method for preparing personalized test sheets are (1) the compilation of relevant item contents and graphical media from a relational database with database queries, (2) the creation of Extensible Markup Language (XML) intermediates, and (3) the transformation into paginated documents.

RESULTS

The software module by use of an open source print formatter consistently produced high-quality test sheets, while the blending of vectorized textual contents and pixel graphics resulted in efficient output file sizes. Concomitantly the module permitted an individual randomization of item sequences to prevent illicit collusion.

CONCLUSIONS

The automatic generation of personalized MCQ test sheets is feasible using freely available open source software libraries, and can be efficiently deployed on a faculty-wide scale.

Aesthetic activities and aesthetic attitudes: Influences of education, background and personality on interest and involvement in the arts

There have been few studies of why some people are frequently involved in aesthetic activities such as going to the theatre, reading or playing musical instruments, whereas others are less involved. This study assesses the broad roles of education, personality and demographic factors such as social class, age and sex. More aesthetic activity was associated with music and art education, whereas science education had a substantial negative relationship with aesthetic activity, both directly and also indirectly via reduced art education. More aesthetic activity was particularly related to higher scores on the personality factor of openness, and also to lower scores on agreeableness and conscientiousness. Higher parental social class was also associated with more aesthetic activity, as also was lower age. Sex had no relationship to aesthetic activity, as neither did masculinity-femininity. Positive aesthetic attitudes were also related moderately to aesthetic activity, but were particularly strongly related to openness to experience, and somewhat less to extraversion. Class, age and sex had no direct relationship to aesthetic attitudes.

Ethical dilemmas in training tomorrow's doctors

This article describes some of the challenges that face trainers and trainees in medical education. There are ethical issues surrounding the use of real patients - particularly children and young people. How much can a young trainee "practise" on a patient? How can they best learn practical skills or gain experience of intimate examinations? The requirements of professional behaviour of trainers towards trainees and of both towards patients are described. Patients involvement in teaching, assessment and formal examinations is discussed.