Creation and Adoption of Large Language Models in Medicine

Importance

There is increased interest in and potential benefits from using large language models (LLMs) in medicine. However, by simply wondering how the LLMs and the applications powered by them will reshape medicine instead of getting actively involved, the agency in shaping how these tools can be used in medicine is lost.

Observations

Applications powered by LLMs are increasingly used to perform medical tasks without the underlying language model being trained on medical records and without verifying their purported benefit in performing those tasks.

Conclusions and Relevance

The creation and use of LLMs in medicine need to be actively shaped by provisioning relevant training data, specifying the desired benefits, and evaluating the benefits via testing in real-world deployments.

ChatGPT Performs on the Chinese National Medical Licensing Examination

ChatGPT, a language model developed by OpenAI, uses a 175 billion parameter Transformer architecture for natural language processing tasks. This study aimed to compare the knowledge and interpretation ability of ChatGPT with those of medical students in China by administering the Chinese National Medical Licensing Examination (NMLE) to both ChatGPT and medical students. We evaluated the performance of ChatGPT in three years' worth of the NMLE, which consists of four units. At the same time, the exam results were compared to those of medical students who had studied for five years at medical colleges. ChatGPT's performance was lower than that of the medical students, and ChatGPT's correct answer rate was related to the year in which the exam questions were released. ChatGPT's knowledge and interpretation ability for the NMLE were not yet comparable to those of medical students in China. It is probable that these abilities will improve through deep learning.

Large language models encode clinical knowledge

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

Medical School Rankings-Bad for the Health of the Profession and the Public

This Viewpoint discusses the limitations of medical school ranking in attracting a diverse student population and urges administrators to holistically communicate their mission, goals, and learning environment as an alternative strategy.

Drawing lessons from the COVID-19 pandemic: science and epistemic humility should go together

During the COVID-19 pandemic, scientific experts advised governments for measures to be promptly taken; they also helped people to understand the situation. They carried out this role in the face of a worldwide emergency, when scientific understanding was still underway. Public scientific disputes also arose, creating confusion among people. This article highlights the importance of experts' epistemic stance under these circumstances. It suggests they should embrace the intellectual virtue of epistemic humility, regulating their epistemic behavior and communication accordingly. In so doing, they would also favour the functioning of the broad network of knowledge-based experts, which is required to properly address all the aspects of the global pandemic.

LifeTime and improving European healthcare through cell-based interceptive medicine

Here we describe the LifeTime Initiative, which aims to track, understand and target human cells during the onset and progression of complex diseases, and to analyse their response to therapy at single-cell resolution. This mission will be implemented through the development, integration and application of single-cell multi-omics and imaging, artificial intelligence and patient-derived experimental disease models during the progression from health to disease. The analysis of large molecular and clinical datasets will identify molecular mechanisms, create predictive computational models of disease progression, and reveal new drug targets and therapies. The timely detection and interception of disease embedded in an ethical and patient-centred vision will be achieved through interactions across academia, hospitals, patient associations, health data management systems and industry. The application of this strategy to key medical challenges in cancer, neurological and neuropsychiatric disorders, and infectious, chronic inflammatory and cardiovascular diseases at the single-cell level will usher in cell-based interceptive medicine in Europe over the next decade.

Medicine's collision with false hope: The False Hope Harms (FHH) argument

The goal of this paper is to introduce the false hope harms (FHH) argument, as a new concept in healthcare. The FHH argument embodies a conglomerate of specific harms that have not convinced providers to stop endorsing false hope. In this paper, it is submitted that the healthcare profession has an obligation to avoid collaborating or participating in, propagating or augmenting false hope in medicine. Although hope serves important functions-it can be 'therapeutic' and important for patients' 'self-identity as active agents'- the presentation of false hope along the hope continuum entails a misconstrued balancing act. By not speaking up against unrealistic patient and family requests-including some requests for rights to try, resuscitative efforts in terminally ill patients, or other demands for non-beneficial treatments-healthcare providers precipitate harms, i.e., the FHH. These harms arise on both individual and communal levels and cannot be ignored. The goal of this paper is not to offer a definition of false hope, because the phenomenon of false hope is too complex for any simple definition. Instead, this paper seeks to make four points while outlining the FHH argument: consumer medicine and false hope are connected; providers and patients are very vulnerable in the system of consumer medicine; providers have a responsibility to stand up against false hope; and how the FHH argument could perhaps offer a footing to resist giving in to false hope.

Conflicts of interest in medicine: a never-ending story

Individuals may have conflicts of interest (CoI) when they choose between the duties of their jobs and their own private interests. In medicine, CoI are potentially ubiquitous and their disclosure has now become the most frequent strategy to address them in professional lives. In the medical literature, CoI are classified into two different types-financial and non-financial. Financial CoI are easy to identify and can bias any kind of results in research; so, their disclosure is very important. The unsolvable dilemma is where to set the lowest limit for sums received from industry. Non-financial CoI are a very large category intrinsically related to the individuals concerned, ranging from family relationships to religious beliefs, and the mere disclosure of many of them can raise privacy and ethical issues. Two opposite narratives characterize the debate on financial CoI caused by pharmaceutical industry. The critical side argues that, because the primary goal of pharmaceutical industry is inevitably to promote its products, the best strategy is to stay away from financial CoI. On the other hand, the defensive side claims that financial CoI are boosted by ideology but meaningless in real practice, since any kind of interest can raise a potential conflict. A missing point in the debate on financial CoI is that health care is a classical example of 'market failure' in the economic theory. Since health cannot be considered a 'consumer good', the economic paradigm of 'free market' does not fit for healthcare products. To conclude, even though transparency on financial CoI cannot itself deter the risk of bias, rejecting it would be an even bigger mistake. At variance, mandatory disclosure of non-financial CoI risks to be confusing and questionable in many cases, paradoxically distracting attention from the potential bias created by financial CoI.

How do trainee doctors learn about research? Content analysis of Australian specialist colleges' intended research curricula

OBJECTIVES

Patients do better in research-intense environments. The importance of research is reflected in the accreditation requirements of Australian clinical specialist colleges. The nature of college-mandated research training has not been systematically explored. We examined the intended research curricula of Australian trainee doctors described by specialist colleges, their constructive alignment and the nature of scholarly project requirements.

DESIGN

We undertook content analysis of publicly available documents to characterise college research training curricula.

SETTING

We reviewed all publicly accessible information from the websites of Australian specialist colleges and their subspecialty divisions. We retrieved curricula, handbooks and assessment-related documents.

PARTICIPANTS

Fifty-eight Australian specialist colleges and their subspecialty divisions.

PRIMARY AND SECONDARY OUTCOME MEASURES

Two reviewers extracted and coded research-related activities as learning outcomes, activities or assessments, by research stage (using, participating in or leading research) and competency based on Bloom's taxonomy (remembering, understanding, applying, analysing, evaluating, creating). We coded learning and assessment activities by type (eg, formal research training, publication) and whether it was linked to a scholarly project. Requirements related to project supervisors' research experience were noted.

RESULTS

Fifty-five of 58 Australian college subspecialty divisions had a scholarly project requirement. Only 11 required formal research training; two required an experienced research supervisor. Colleges emphasised a role for trainees in leading research in their learning outcomes and assessments, but not learning activities. Less emphasis was placed on using research, and almost no emphasis on participation. Most learning activities and assessments mapped to the 'creating' domain of Bloom's taxonomy, whereas most learning outcomes mapped to the 'evaluating' domain. Overall, most research learning and assessment activities were related to leading a scholarly project.

CONCLUSIONS

Australian specialist college research curricula appear to emphasise a role for trainees in leading research and producing research deliverables, but do not mandate formal research training and supervision by experienced researchers.

When Physicians Don't Know

Given the boundless amount of scientific information, clinical skills, and interventional techniques present in biomedicine today, it is impossible for individual physicians and clinicians to have absolute medical knowledge. Further, ambiguity in the interpretation and treatment of illness can lead to significant uncertainty. Despite the inevitability of not knowing in biomedicine, however, there is relatively little academic discussion about how physicians are socialized to address ignorance, how clinicians experience gaps in knowledge as practitioners, or the various forms that not knowing takes in professional health-care practice and education. This article seeks to invigorate new discussions on the role of ignorance and "non-knowledge" in biomedical practice and training. The article critically examines the predominant focus on medical knowledge in the sociological literature and presents a new anthropological framework for the relationship between knowing and not knowing in medicine, called "sufficient knowledge." The author posits that future social and humanistic examinations of biomedicine should seriously consider the ways that physicians navigate ignorance, uncertainty, and not knowing, and that scientists, clinicians, social scientists, and ethicists all have valuable disciplinary perspectives to bring to the conversation around medical ignorance.

Assessment methods in medical specialist assessments in the DACH region - overview, critical examination and recommendations for further development

Introduction: Specialist medical assessments fulfil the task of ensuring that physicians have the clinical competence to independently represent their field and provide the best possible care to patients, taking into account the current state of knowledge. To date, there are no comprehensive reports on the status of specialist assessments in the German-speaking countries (DACH). For that reason, the assessment methods used in the DACH region are compiled and critically evaluated in this article, and recommendations for further development are described. Methods: The websites of the following institutions were searched for information regarding testing methods used and the organisation of specialist examinations: Homepage of the Swiss Institute for Medical Continuing Education (SIWF), Homepage of the Academy of Physicians (Austria) and Homepage of the German Federal Medical Association (BAEK). Further links were considered and the results were presented in tabular form. The assessment methods used in the specialist assessments are critically examined with regard to established quality criteria and recommendations for the further development of the specialist assessments are derived from these. Results: The following assessment methods are already used in Switzerland and Austria: written examinations with multiple choice and short answer questions, structured oral examinations, the Script Concordance Test (SCT) and the Objective Structured Clinical Examination (OSCE). In some cases, these assessment methods are combined (triangulation). In Germany, on the other hand, the oral examination has so far been conducted in an unstructured manner in the form of a 'collegial content discussion'. In order to test knowledge, practical and communicative competences equally, it is recommended to implement a triangulation of methods and follow the further recommendations described in this article. Conclusion: While there are already accepted approaches for quality-assured and competence-based specialist assessments in Switzerland and Austria at present, there is still a long way to go in Germany. Following the recommendations presented in this article, a contribution could be made to improving the specialist assessments in the DACH region according to the specialist assessments objectives.

A systematic review of quality of life research in medicine and health sciences

Purpose

Quality of life (QOL) is an important concept in the field of health and medicine. QOL is a complex concept that is interpreted and defined differently within and between disciplines, including the fields of health and medicine. The aims of this study were to systematically review the literature on QOL in medicine and health research and to describe the country of origin, target groups, instruments, design, and conceptual issues.

Methods

A systematic review was conducted to identify research studies on QOL and health-related quality of life (HRQOL). The databases Scopus, which includes Embase and MEDLINE, CINAHL, and PsycINFO were searched for articles published during one random week in November 2016. The ten predefined criteria of Gill and Feinstein were used to evaluate the conceptual and methodological rigor.

Results

QOL research is international and involves a variety of target groups, research designs, and QOL measures. According to the criteria of Gill and Feinstein, the results show that only 13% provided a definition of QOL, 6% distinguished QOL from HRQOL. The most frequently fulfilled criteria were: (i) stating the domains of QOL to be measured; (ii) giving a reason for choosing the instruments used; and (iii) aggregating the results from multiple items.

Conclusion

QOL is an important endpoint in medical and health research, and QOL research involves a variety of patient groups and different research designs. Based on the current evaluation of the methodological and conceptual clarity of QOL research, we conclude that the majority QOL studies in health and medicine have conceptual and methodological challenges.

Electronic supplementary material

The online version of this article (doi:10.1007/s11136-019-02214-9) contains supplementary material, which is available to authorized users.

Power of Judgment: The Significance of Kant's Philosophy for the Medical System Today

The ways of thinking in the manufacturing sciences are increasingly determining the rationality within medicine as a practical or action-based science. This "technological paradigm" infiltrates the field of medicine with the promise of increasing efficiency while simultaneously improving quality at various points in the system. Simple linear causal relationships generally need to be taken into account when manufacturing products. Even complex manufacturing processes can be broken down into the smallest units and, therefore, also be automated. The situation in complex systems such as the human body, however, is completely different. In order for doctors to be able to carry out their actions within this complex system, medicine as a science provides the physician with rules on the means that should be used to decide which remedy should be used, when and how. This judgment of which remedy should be used, when and how, what is known as the indication, is a central medical moment. This requires a power of judgment sharpened by experience. The indication, in turn, essentially determines the course of a disease and thus the quality of the treatment or the quality of result so often referred to these days.

Rethinking medicine

There’s something going on out there

Better medicine through machine learning: What's real, and what's artificial?

Machine Learning Special Issue Guest Editors Suchi Saria, Atul Butte, and Aziz Sheikh cut through the hyperbole with an accessible and accurate portrayal of the forefront of machine learning in clinical translation.

[Evaluation of The Medical Board Exam in Portugal]

INTRODUCTION

There is a high heterogeneity in the structure of postgraduate medical training evaluation worldwide. However, in contrast to other countries, there have been no scientific studies of the final medical board examination, in Portugal. The present study aimed to evaluate the adequacy of the medical board examination including its validity as measured by its association with medical school grade average and national seriation examination.

MATERIAL AND METHODS

Cross-sectional, observational study. We analyzed the final results on the medical board examination of 2439 physicians, across 47 specialties, who completed their training in 2016 and 2017, using measures of central tendency and variability. We assessed the association between these grades and the national exam to initiate residency, and the grade average in Medical School.

RESULTS

Measures of central tendency and variability, and consequent shape measures, revealed that the distribution of the scores of the final medical board exam is extremely negatively asymmetric and leptokurtic. A positive association was also found between the results in this exam and the score on national exam to initiate residency, and the grade average in Medical School.

CONCLUSION

Although the medical board examination was, in general, positively associated with scores on the national exam to initiate residency, and the mean final Medical School grades, thus indicating its potential validity, our results demonstrate that this exam presents no satisfactory discriminative capacity. Therefore, there is room to improve the actual postgraduate medical examination model, including changes in its classification system and potentially consider other assessment models.

Students with an autonomous role in hospital care - patients perceptions

BACKGROUND

There have been calls to enhance clinical education by strengthening supported active participation (SAP) of medical students in patient care. This study examines perceived quality of care when final-year medical students are integrated in hospital ward teams with an autonomous relationship toward their patients.

METHODS

We established three clinical education wards (CEWs) where final-year medical students were acting as "physician under supervision". A questionnaire-based mixed-method study of discharged patients was completed in 2009-15 using the Picker Inpatient Questionnaire complemented by specific questions on the impact of SAP. Results were compared with matched pairs of the same clinical specialty from the same hospital (CG1) and from nationwide hospitals (CG2). Patients free-text feedback about their hospital stay was qualitatively evaluated.

RESULTS

Of 1136 patients surveyed, 528 (46.2%) returned the questionnaire. The CEWs were highly recommended, with good overall quality of care and patient-physician/student-interaction, all being significantly (p < 0.001) higher for the CEW group while experienced medical treatment success was similar. Patient-centeredness of students was appreciated by patients as a support to a deeper understanding of their condition and treatment.

CONCLUSION

Our study indicates that SAP of final-year medical students is appreciated by patients with high overall quality of care and patient-centeredness.

Longitudinal assessments in continuing specialty certification and lifelong learning

Longitudinal assessment (LA) involves the regular, spaced delivery of a limited number of questions on practice relevant content on a computer or mobile internet platform. Depending on the platform, participants may indicate relevance of the content to their practice and confidence in their answer prior to receiving immediate feedback (including critiques) on each question. Individual dashboards may be included to assist participants in tracking progress and identifying areas of strength and weaknesss across a content blueprint. This paper provides an overview of the theoretical underpinnings underlying LA programs, briefly describes current uses of LA in medicine and suggests areas for evaluating the role of LA in continuing medical specialty certification and continuing professional development.

Quality of medicines: Deficiencies found by Brazilian Health Regulatory Agency (ANVISA) on good manufacturing practices international inspections

The circulation of poor quality medicines, especially in the developing countries, is a public health concern. Compliance with good manufacturing practices (GMP) is essential to ensure the quality, efficacy, and safety of medicines. This study evaluated the outcomes of the Brazilian Health Regulatory Agency's (ANVISA) international inspections of two years (2015 and 2016) and compared these to those of other regulatory authorities. The information from 255 inspection reports was analyzed, and the type and extent of deficiencies were collected. In the period evaluated, 62.75% of ANVISA-inspected companies were classified as GMP "satisfactory," 24.71% were classified as having "on demand" status, and 12.55% of inspections concluded that the company did not comply with Brazilian GMP regulations ("unsatisfactory"). The most common areas of deficiency were documentation (28.63%) and premises (26.27%). The pattern of deficiencies was similar to the findings of other regulatory agencies. However, ANVISA detected a more significant number of non-compliance results than other authorities, which may be caused by differences in classifications adopted by each Agency. Furthermore, manufacturers inspected by ANVISA may follow different standards and practices for products manufactured for the Brazilian market. Disclosure of main GMP deficiencies found can be useful for encouraging the industry to comply with GMP, and additional guidelines in the specific areas where deficiencies are often identified may be useful to industry to improve GMP compliance. Harmonization of GMP guidelines and inspection procedures are the key steps to avoid duplicate work, but regulatory authorities also need to work together to enforce the proper level of GMP compliance by pharmaceutical manufacturers, assuring high quality and safe medicines supply.

Military Medicine Implements In-home Virtual Health in Europe

OBJECTIVE

This report outlines a multispecialty implementation effort which included 12 specialty practices and 28 clinicians within Regional Health Command Europe (RHCE) and Landstuhl Regional Medical Center (LRMC) to pilot an in-home virtual health (VH) program using existing resources.

METHODS AND MATERIALS

Synchronous VH encounters were performed using an Acano desktop conferencing client (Cisco Systems, Inc, San Jose, CA) and a USB web camera at the provider (distant) site and the patient's own computer or device in the home. A web real-time conferencing (Web RTC) server provided the connections.

RESULTS

Between October 2016 and May 2018, 310 synchronous VH appointments to patients' homes in 23 geographic locations in 9 countries on 3 different continents were completed; 28 skill type I and II specialty providers at LRMC, SHAPE Belgium Army Health Clinic (AHC), and Vilseck AHC, Germany Primary Care Clinic participated. The providers represented 9 distinct specialties and primary care. Appointment types were as follows: 85 (39%) follow-up type appointments; 70 (32%) group type appointments; 65 (30%) initial specialty care appointments. The 3 most active clinics were Pediatric Gastroenterology with 88 (28%), the Nutrition Clinic with 82 (26%), and the Traumatic Brain Injury Clinic with 63 (20%) encounters. Full audio and video connectivity rate was 97%, excluding reconnects after dropped calls which occasionally occurred. Patient satisfaction scores were high 16/17 (94%) with 5% of patients surveyed.

CONCLUSION

Low complexity synchronous VH appointments were successfully accomplished across a broad spectrum of health care services and appointment types. Landstuhl RMC specialists received consults from sites across a vast geographic area including Europe, the Middle East, and Africa. An in-home VH option gives providers a special tool to extend services far beyond traditional boundaries. This pilot project helped RHCE and LRMC providers gain valuable experience extending care to the home and will provide foundational knowledge for future VH efforts targeting groups and outcomes.

Defining and Measuring Diagnostic Uncertainty in Medicine: A Systematic Review

BACKGROUND

Physicians routinely encounter diagnostic uncertainty in practice. Despite its impact on health care utilization, costs and error, measurement of diagnostic uncertainty is poorly understood. We conducted a systematic review to describe how diagnostic uncertainty is defined and measured in medical practice.

METHODS

We searched OVID Medline and PsycINFO databases from inception until May 2017 using a combination of keywords and Medical Subject Headings (MeSH). Additional search strategies included manual review of references identified in the primary search, use of a topic-specific database (AHRQ-PSNet) and expert input. We specifically focused on articles that (1) defined diagnostic uncertainty; (2) conceptualized diagnostic uncertainty in terms of its sources, complexity of its attributes or strategies for managing it; or (3) attempted to measure diagnostic uncertainty.

KEY RESULTS

We identified 123 articles for full review, none of which defined diagnostic uncertainty. Three attributes of diagnostic uncertainty were relevant for measurement: (1) it is a subjective perception experienced by the clinician; (2) it has the potential to impact diagnostic evaluation-for example, when inappropriately managed, it can lead to diagnostic delays; and (3) it is dynamic in nature, changing with time. Current methods for measuring diagnostic uncertainty in medical practice include: (1) asking clinicians about their perception of uncertainty (surveys and qualitative interviews), (2) evaluating the patient-clinician encounter (such as by reviews of medical records, transcripts of patient-clinician communication and observation), and (3) experimental techniques (patient vignette studies).

CONCLUSIONS

The term "diagnostic uncertainty" lacks a clear definition, and there is no comprehensive framework for its measurement in medical practice. Based on review findings, we propose that diagnostic uncertainty be defined as a "subjective perception of an inability to provide an accurate explanation of the patient's health problem." Methodological advancements in measuring diagnostic uncertainty can improve our understanding of diagnostic decision-making and inform interventions to reduce diagnostic errors and overuse of health care resources.

How do national specialty groups develop undergraduate guidelines for medical schools, and which are successful? A systematic review

PURPOSE

To better understand the steps undertaken by medical specialties to develop and implement undergraduate national, and international, educational guidelines for use in medical schools, and to find what makes them successful in terms of uptake and knowledge.

METHODS

Systematic review of databases to find inter- and nationally-created undergraduate medical specialty guidelines, and descriptions of development and analysis, from 1998 to January 2015.

RESULTS

Ninety six eligible papers were found, covering 59 different guidelines in 32 specialties. Five documented from development to revision. Development often required multiple stages and methods, 10 using the Delphi technique. Twenty two guidelines mapped to recommended government standards. Twenty papers analyzed curricula. No guideline was used in every relevant medical school.

CONCLUSIONS

This is a comprehensive review of the processes involved in creating international and national guidelines, with emphasis of key points for those considering similar undertakings. These include thorough needs analysis of multiple groups involved in the delivery of the curriculum; and engagement of relevant parties throughout development, to ensure relevance and increase buy-in. Flexibility is important, to allow use in medical schools with different methods of teaching. Ongoing evaluation and update are also critical steps that must not be forgotten.

23rd Gordon Arthur Ransome Oration: Law and Medicine: Professions of Honour, Service and Excellence

Amidst dramatic global change, there is a greater need than ever for principled and committed professionals. The sibling professions of law and medicine, in particular, serve crucial functions in contemporary society. To meet the challenges of the future, members of both professions must appreciate and commit to their shared calling to public service, and ensure that it is not overridden by the profit motive. Three further commitments follow from this: first, to nurture and mentor new entrants to the professions to ensure the continuance of that mission of public service; second, to improve accessibility and affordability through pro bono work and by other means, so that all who need professional assistance can receive it; and, thirdly, to serve with excellence in ethics as well as competence, so as not to betray the trust of the laypersons who rely on professionals to safeguard their interests and welfare. The medical and legal professions have much to learn and gain from working with each other toward those shared goals.

Standardisation of delivery and assessment of research training for specialty trainees based on curriculum requirements: recommendations based on a scoping review

OBJECTIVES

(1) To conduct a scoping review of postgraduate specialty training (ST) curricula for doctors within Health Education England in order to identify common themes and variations in requirements for training and assessment of research competencies. (2) To make recommendations on standardisation of training for clinical research across ST programmes.

SETTING

Health Education England North East and National Institute for Health Research Clinical Research Network (CRN)-North East and North Cumbria.

METHODS

Annual Review of Competence Progression (ARCP); Certificate of Completion of Training (CCT) checklists and curricula for ST were obtained from Health Education England North East and reviewed between June and September 2015. Research competence requirements based on knowledge, skills or behaviour-based domains were identified and entered onto a spreadsheet for analysis. Common themes with levels of competence required were identified. This information was used to construct and propose a model for delivery of training in clinical research across ST programmes.

RESULTS

Sixty-two ST curricula were reviewed and seven common themes for research training were found in up to 97% of the curricula. Requirement for good clinical practice (GCP) in research training was included in 15% of curricula. One of the common themes involved knowledge-based competency, and three each of the remaining seven involved skills or behaviour-based competencies. There was less clarity and larger variation between specialties in how research competencies were assessed; and what evidence was required for ARCP and CCT to assure competence. 63% (19/30) of curricula from medical specialties had no mention of research requirements within their ARCP guidelines.

CONCLUSIONS

Given that the majority of specialty curricula contain consistent themes around core research knowledge, consideration should be given to standardising the delivery and assessment of generic research competencies within ST. Our recommendations from this review could form the basis for developing structured research training for specialty trainees involving: (1) a taught course for knowledge-based competencies; (2) clinical placements with CRN teams for practical workplace-based experience and (3) developing research tutors to help support placements and assessment of these competencies.

The predictive validity of a situational judgement test, a clinical problem solving test and the core medical training selection methods for performance in specialty training

The aim of this study was to follow up a sample of physicians who began core medical training (CMT) in 2009. This paper examines the long-term validity of CMT and GP selection methods in predicting performance in the Membership of Royal College of Physicians (MRCP(UK)) examinations. We performed a longitudinal study, examining the extent to which the GP and CMT selection methods (T1) predict performance in the MRCP(UK) examinations (T2). A total of 2,569 applicants from 2008-09 who completed CMT and GP selection methods were included in the study. Looking at MRCP(UK) part 1, part 2 written and PACES scores, both CMT and GP selection methods show evidence of predictive validity for the outcome variables, and hierarchical regressions show the GP methods add significant value to the CMT selection process. CMT selection methods predict performance in important outcomes and have good evidence of validity; the GP methods may have an additional role alongside the CMT selection methods.

[Assessments during Medical Specialists Training: quantity or quality?]

Structured assessments form a mandatory part of Dutch Medical Specialist Training, but create administrative workload for both the staff and supervisors. One could argue that the quality of the narrative feedback is more important than the extensive reporting in learning portfolios, and that the focus should be on continuous on-the-job coaching.