Dutch postgraduate GP selection procedure; reliability of interview assessments

Evidence-Based Practices for Interviewing Graduate Medical Education Applicants: A Systematic Review

Background Although the selection interview is a standard admission practice for graduate medical education (GME) programs in the United States, there is a dearth of recent reviews on optimizing the trainee interview process, which has low reliability, high cost, and major risk of bias. Objective To investigate the evidence base for different selection interview practices in GME. Methods We searched 4 literature databases from inception through September 2022. Two investigators independently conducted title/abstract screening, full-text review, data extraction, and quality assessment. Disagreements were mediated by discussion. We used backward reference searching of included articles to identify additional studies. We included studies of different interview methods and excluded literature reviews, non-GME related publications, and studies comparing different applicant populations. We examined study characteristics, applicant and interviewer preferences, and interview format. We evaluated study quality using the Medical Education Research Study Quality Instrument (MERSQI). Results Of 2192 studies, 39 (2%) met our inclusion criteria. The evidence base was rated as moderately low quality using MERSQI criteria. Applicants reported preferences for several one-on-one interviews lasting 15 to 20 minutes, interviews by current trainees, and interviews including social events with only trainees. Applicants had mixed perceptions of virtual versus in-person interviews and reported that virtual interviews saved costs. The multiple mini interview (MMI) required more applicant and interviewer time than individual interviews but demonstrated construct and predictive validity and was preferred by applicants and interviewers. Conclusions Based on moderately low-quality evidence, using the MMI, training interviewers, and providing applicants with basic program information in advance should be considered for GME selection interviews.

Medical student selection process enhanced by improving selection algorithms and changing the focus of interviews in Australia: a descriptive study

PURPOSE

The study investigates the efficacy of new features introduced to the selection process for medical school at the University of New South Wales, Australia: (1) considering the relative ranks rather than scores of the Undergraduate Medicine and Health Sciences Admission Test and Australian Tertiary Admission Rank; (2) structured interview focusing on interpersonal interaction and concerns should the applicants become students; and (3) embracing interviewers’ diverse perspectives.

METHODS

Data from 5 cohorts of students were analyzed, comparing outcomes of the second year in the medicine program of 4 cohorts of the old selection process and 1 of the new process. The main analysis comprised multiple linear regression models for predicting academic, clinical, and professional outcomes, by section tools and demographic variables.

RESULTS

Selection interview marks from the new interview (512 applicants, 2 interviewers each) were analyzed for inter-rater reliability, which identified a high level of agreement (kappa=0.639). No such analysis was possible for the old interview since it required interviewers to reach a consensus. Multivariate linear regression models utilizing outcomes for 5 cohorts (N=905) revealed that the new selection process was much more effective in predicting academic and clinical achievement in the program (R2=9.4%–17.8% vs. R2=1.5%–8.4%).

CONCLUSION

The results suggest that the medical student selection process can be significantly enhanced by employing a non-compensatory selection algorithm; and using a structured interview focusing on interpersonal interaction and concerns should the applicants become students; as well as embracing interviewers’ diverse perspectives.

Development and validation of a new instrument measuring guideline adherence in clinical practice

BACKGROUND

Education in evidence-based medicine (EBM) is an important part of the postgraduate training of GPs. Evaluation of its effect on EBM behaviour in daily clinical practice is difficult and instruments are scarce. Working in accordance with guidelines is considered as one of the key indicators of EBM behaviour.

OBJECTIVE

To develop and validate an instrument assessing guideline adherence of GP trainees in clinical practice.

METHODS

We developed an instrument that assesses guideline adherence, taking conscious deviation into account. The instrument assesses guideline adherence on 59 different management decisions (diagnosis N = 17, therapy N = 20, referral N = 22) for 23 conditions as described in 27 different clinical practice guidelines. We validated this instrument using performance data as collected by third-year GP trainees on three important properties: validity, reliability and feasibility.

RESULTS

Performance data were collected by 76 GP trainees on 12106 patient consultations with 12587 different reasons for encounter. Overall, guideline adherence was 82% (95% confidence interval 77-88%). The significant correlation with the national GP knowledge test (r 0.33, P 0.004) showed the instrument to be a valid instrument. Interrater reliabilities (intraclass correlation coefficient) varied between moderate and excellent (0.64-1.00, P < 0.001). The instrument proved feasible with coverage of 24% (N = 3082) of reasons for encounter presented to GP trainees and a mean and median time of 1 minute to score a patient consultation.

CONCLUSION

This instrument proved valid, reliable and feasible to assess guideline adherence among trainees in the clinical primary care setting.

What predicts doctors' satisfaction with their chosen medical specialty? A Finnish national study

BACKGROUND

In Finland the number of medical specialists varies between specialties and regions. More regulation of the post-graduate medical training is planned. Therefore, it is important to clarify what predicts doctors' satisfaction with their chosen specialty.

METHODS

A random sample contained 50% of all Finnish doctors under 70 years of age. The respose rate was 50.5%. Working-age specialists were asked to value their motives when choosing a specialty. They were also asked if they would choose the same specialty again. The odds ratios for not choosing the same specialty again were tested.

RESULTS

Diversity of work was the most important motive (74% of respondents). Seventeen percent of GPs would not choose the same specialty again, compared to 2% of ophthalmologists and 4% of pediatricians. A major role of Diversity of work and Prestigious field correlated with satisfaction whereas Chance with dissatisfaction with the specialty.

DISCUSSION

Motives and issues related to the work and training best correlate with satisfaction with the specialty.

CONCLUSIONS

When the numbers of Finnish postgraduate medical training posts become regulated, a renewed focus should be given to finding the most suitable speciality for each doctor. Information about employment and career advice should play an important role in this.

A competency based selection procedure for Dutch postgraduate GP training: a pilot study on validity and reliability

Abstract Background: Historically, semi-structured interviews (SSI) have been the core of the Dutch selection for postgraduate general practice (GP) training. This paper describes a pilot study on a newly designed competency-based selection procedure that assesses whether candidates have the competencies that are required to complete GP training.

OBJECTIVES

The objective was to explore reliability and validity aspects of the instruments developed.

METHODS

The new selection procedure comprising the National GP Knowledge Test (LHK), a situational judgement tests (SJT), a patterned behaviour descriptive interview (PBDI) and a simulated encounter (SIM) was piloted alongside the current procedure. Forty-seven candidates volunteered in both procedures. Admission decision was based on the results of the current procedure.

RESULTS

Study participants did hardly differ from the other candidates. The mean scores of the candidates on the LHK and SJT were 21.9 % (SD 8.7) and 83.8% (SD 3.1), respectively. The mean self-reported competency scores (PBDI) were higher than the observed competencies (SIM): 3.7(SD 0.5) and 2.9(SD 0.6), respectively. Content-related competencies showed low correlations with one another when measured with different instruments, whereas more diverse competencies measured by a single instrument showed strong to moderate correlations. Moreover, a moderate correlation between LHK and SJT was found. The internal consistencies (intraclass correlation, ICC) of LHK and SJT were poor while the ICC of PBDI and SIM showed acceptable levels of reliability.

CONCLUSION

Findings on content validity and reliability of these new instruments are promising to realize a competency based procedure. Further development of the instruments and research on predictive validity should be pursued.