A general practice workplace-based assessment instrument: Content and construct validity

Prediction of general practice licensure/certification examination performance by a measure of 'questionable' activities in clinical practice: a retrospective cohort study

PURPOSE

'Low-value' clinical care and medical services are 'questionable' activities, being more likely to cause harm than good or with disproportionately low benefit relative to cost. This study examined the predictive ability of the QUestionable In Training Clinical Activities Index (QUIT-CAI) for general practice (GP) registrars' (trainees') performance in Australian GP Fellowship examinations (licensure/certification examinations for independent GP).

METHODS

The study was nested in ReCEnT, an ongoing cohort study in which Australian GP registrars document their in-consultation clinical practice. Outcome factors in analyses were individual registrars' scores on the three Fellowship examinations ('AKT', 'KFP', and 'OSCE' examinations) and pass/fail rates during 2012-21. Analyses used univariable and multivariable regression (linear or logistic, as appropriate). The study factor in each analysis was 'QUIT-CAI score percentage'-the percentage of times a registrar performed a QUIT-CAI clinical activity when 'at risk' (i.e. when managing a problem where performing a QUIT-CAI activity was a plausible option).

RESULTS

A total of 1265, 1145, and 553 registrars sat Applied Knowledge Test, Key Features Problem, and Objective Structured Clinical Exam examinations, respectively. On multivariable analysis, higher QUIT-CAI score percentages (more questionable activities) were significantly associated with poorer Applied Knowledge Test scores (P = .001), poorer Key Features Problem scores (P = .003), and poorer Objective Structured Clinical Exam scores (P = .005). QUIT-CAI score percentages predicted Royal Australian College of General Practitioner exam failure [odds ratio 1.06 (95% CI 1.00, 1.12) per 1% increase in QUIT-CAI, P = .043].

CONCLUSION

Performing questionable clinical activities predicted poorer performance in the summative Fellowship examinations, thereby validating these examinations as measures of actual clinical performance (by our measure of clinical performance, which is relevant for a licensure/certification examination).

Establishment of an Evaluation Index System of Competencies for College Senior Students in General Practice Medicine in Anhui Province, China

Background

The competencies of college senior students in general practice medicine have attracted attention. This study aimed to construct an evaluation index system of competencies for college senior students in general practice medicine and to promote the reform and optimization of training programs for general medicine talent in colleges.

Methods

The two-round Delphi method was used to determine the evaluation index system of competencies for college senior students in general practice medicine, and the analytic hierarchy process (AHP) was used to calculate the weights of all levels of elements.

Results

The evaluation index system of competencies for college senior students in general practice medicine was established with 3 primary factors, 9 secondary factors and 32 tertiary factors. The Delphi results revealed that the active coefficient of experts was 1 and the authority coefficient was 0.858. The 3 primary factors were knowledge level, job skills and professionalism with weights of 0.1532, 0.4207 and 0.4261, respectively. Among the secondary factors, the top three weight coefficients were professional ethics (0.2614), community practice (0.1526) and communication skills (0.1308). Among tertiary factors, "scientific research" exhibited the lowest value with a weight coefficient of 0.0049.

Conclusion

In this study, we constructed an evaluation index system of competencies for college senior students in general practice medicine. The consensus on the content of the competencies of college senior students in general practice medicine suggests that these elements are necessary for those who will become general practitioners. This system can be used as the basis to evaluate the ability of college senior students in general practice medicine and provide guidance for the cultivation and evaluation of general medicine talent.

Exam prediction and the general Practice Registrar Competency Assessment Grid (GPR-CAG)

BACKGROUND

In GP training, identifying early predictors of poor summative examination performance can be challenging. We aimed to establish whether external clinical teaching visit (ECTV) performance, measured using a validated instrument (GP Registrar Competency Assessment Grid, GPR-CAG) is predictive of Royal Australian College of General Practitioners (RACGP) Fellowship examination performance.

METHODS

A retrospective cohort study including GP registrars in New South Wales/Australian Capital Territory with ECTV data recorded during their first training term (GPT1), between 2014 and 2018, who attempted at least one Fellowship examination. Independent variables of interest included the four GPR-CAG factors assessed in GPT1 ('patient-centredness/caring', 'formulating hypotheses/management plans', 'professional responsibilities', 'physical examination skills'). Outcomes of interest included individual scores of the three summative examinations (Applied Knowledge Test (AKT); Key Feature Problem (KFP); and the Objective Structured Clinical Examination (OSCE)) and overall Pass/Fail status. Univariable and multivariable regression analyses were performed.

RESULTS

Univariably, there were statistically significant associations (p < 0.01) between all four GPR-CAG factors and all four summative examination outcomes, except for 'formulating hypotheses/management plans' and OSCE score (p = 0.07). On multivariable analysis, each factor was significantly associated (p < 0.05) with at least one exam outcome, and 'physical examination skills' was significantly associated (p < 0.05) with all four exam outcomes.

DISCUSSION

ECTV performance, via GPR-CAG scores, is predictive of RACGP Fellowship exam performance. The univariable findings highlight the pragmatic utility of ECTVs in flagging registrars who are at-risk of poor exam performance, facilitating early intervention. The multivariable associations of GPR-CAG scores and examination performance suggest that these scores provide predictive ability beyond that of other known predictors.

The Registrar Clinical Encounters in Training (ReCEnT) cohort study: updated protocol

BACKGROUND

During vocational general practice training, the content of each trainee's (in Australia, registrars') in-consultation clinical experience is expected to entail a breadth of conditions that exemplify general practice, enabling registrars to gain competency in managing common clinical conditions and common clinical scenarios. Prior to the Registrar Clinical Encounters in Training (ReCEnT) project there was little research into the content of registrars' consultations despite its importance to quality of training. ReCEnT aims to document the consultation-based clinical and educational experiences of individual Australian registrars.

METHODS

ReCEnT is an inception cohort study. It is comprised of closely interrelated research and educational components. Registrars are recruited by participating general practice regional training organisations. They provide demographic information about themselves, their skills, and their previous training. In each of three 6-month long general practice training terms they provide data about the practice where they work and collect data from 60 consecutive patient encounters using an online portal. Analysis of data uses standard techniques including linear and logistic regression modelling. The ReCEnT project has approval from the University of Newcastle Human Research Ethics Committee, Reference H-2009-0323.

DISCUSSION

Strengths of the study are the granular detail of clinical practice relating to patient demographics, presenting problems/diagnoses, medication decisions, investigations requested, referrals made, procedures undertaken, follow-up arranged, learning goals generated, and in-consultation help sought; the linking of the above variables to the presenting problems/diagnoses to which they pertain; and a very high response rate. The study is limited by not having information regarding severity of illness, medical history of the patient, full medication regimens, or patient compliance to clinical decisions made at the consultation. Data is analysed using standard techniques to answer research questions that can be categorised as: mapping analyses of clinical exposure; exploratory analyses of associations of clinical exposure; mapping and exploratory analyses of educational actions; mapping and exploratory analyses of other outcomes; longitudinal 'within-registrar' analyses; longitudinal 'within-program' analyses; testing efficacy of educational interventions; and analyses of ReCEnT data together with data from other sources. The study enables identification of training needs and translation of subsequent evidence-based educational innovations into specialist training of general practitioners.

'Low-value' clinical care in general practice: associations of low value care in GP trainees' practice, including formative and summative examination performance - protocol for cross-sectional and retrospective cohort study analyses using the QUestionable In Training Clinical Activities (QUIT-CA) index

INTRODUCTION

'Low value' clinical care and overuse of medical services are 'questionable' clinical activities that entail provision of medical services that are more likely to cause harm than good or whose benefit is disproportionately low compared with its cost. This study will seek to establish clinical practice associations of a non-observed work-based assessment of general practitioner (GP) trainees' (registrars') questionable practice (the QUestionable In Training Clinical Activities (QUIT-CA) index). We will also explore association of the QUIT-CA index with a formative observed work-based assessment, and will establish if registrars' QUIT-CA indexes are associated with summative examination performance.

METHODS AND ANALYSIS

We will conduct three analyses, all using data from the Registrar Clinical Encounters in Training (ReCEnT) study. ReCEnT is an ongoing (from 2010) cohort study in which Australian GP registrars record details of their in-consultation clinical and educational practice. The QUIT-CA index is compiled from ReCEnT consultation data. A cross-sectional analysis, using negative binomial regression, will establish clinical practice associations of the QUIT-CA index. A cross-sectional analysis using linear regression will be used to establish associations of QUIT-CA index with formative observed in-practice assessment (the General Practice Registrar-Competency Assessment Grid). A retrospective cohort study analysis using linear regression will be used to establish associations of the QUIT-CA index with summative examination performance (Royal Australian College of General Practice fellowship examinations results).

ETHICS AND DISSEMINATION

The study has ethical approval from the University of Newcastle HREC(H-2009-0323). Findings will be disseminated in peer-reviewed journal articles and conference presentations.

Study protocol: content and perceived educational utility of different modalities of clinical teaching visit (CTV) workplace-based assessments within Australian general practice vocational training: a cross-sectional study

INTRODUCTION

Clinical teaching visits (CTVs) are formative workplace-based assessments that involve a senior general practitioner (GP) observing a clinical practice session of a general practice registrar (specialist vocational GP trainee). These visits constitute a key part of Australian GP training. Despite being mandatory and resource-intensive, there is a paucity of evidence regarding the content and educational utility of CTVs. This study aims to establish the content and educational utility of CTVs across varying practice settings within Australia, as perceived by registrars and their assessors ('CT visitors'). In addition, this study aims to establish registrar, CT visitor and practice factors associated with CTV content and perceived CTV utility ratings.

METHODS AND ANALYSIS

This study will collect data prospectively using online questionnaires completed soon after incident CTVs. Participants will be registrars and CT visitors of CTVs conducted from March 2020 to January 2021. The setting is three Regional Training Organisations across four Australian states and territories (encompassing 37% of Australian GP registrars).Outcome factors will be a number of specified CTV content elements occurring during the CTV as well as participants' perceptions of CTV utility, which will be analysed using univariate and multivariable regression.

ETHICS AND DISSEMINATION

Ethics approval has been granted by the University of Newcastle Human Research Ethics Committee, approval number H-2020-0037. Study findings are planned to be disseminated via conference presentation, peer-reviewed journals, educational practice translational workshops and the GP Synergy research subwebsite.