Relationship of electronic medical knowledge resource use and practice characteristics with Internal Medicine Maintenance of Certification Examination scores

Feasibility, acceptability and impact of a clinical decision support tool among primary care providers in an urban, rural and remote site in the Philippines

INTRODUCTION

Strengthening primary care helps address health inequities that continue to persist in the Philippines. The Philippine Primary Care Studies pilot-tested interventions to improve the primary care system. One intervention was the provision of a free subscription to an electronic decision support application called UpToDate (UTD) for primary care providers (PCPs), including doctors, nurses, midwives and community health workers (CHWs). This study aimed to (1) assess the feasibility of using UTD as information source for PCPs in urban, rural and remote settings, (2) determine the acceptability of UTD as an information source for PCPs and (3) examine the impact of UTD access on PCP clinical decision-making.

METHODS

Four focus group discussions (FGDs) and two key informant interviews (KII) were conducted to gather insights from 30 PCPs. Thematic analysis through coding in NVivo V.12 was done using the technology acceptance model (TAM) as a guiding framework.

RESULTS

All PCPs had positive feedback regarding UTD use because of its comprehensiveness, accessibility, mobility and general design. The participants relayed UTD's benefit for point-of-contact use, capacity-building and continuing professional development. PCPs across the three sites, including CHWs with no formal medical education, were able to provide evidence-based medical advice to patients through UTD. However, external factors in these settings impeded the full integration of UTD in the PCPs' workflow, including poor internet access, unstable sources of electricity, lack of compatible mobile devices and the need for translation to the local language.

CONCLUSION

UTD was a feasible and acceptable clinical decision support tool for the PCPs. Factors affecting the feasibility of using UTD include technological and environmental factors (ie, internet access and the lack of translation to the local language), as well as the organisational structure of the primary care facility which determines the roles of the PCPs. Despite the difference in roles and responsibilities of the PCPs, UTD positively impacted decision-making and patient education for all PCPs through its use as a point-of-contact tool and a tool for capacity-building.

Scientific literacy and preferred resources used by Latin American medical students during the COVID-19 pandemic: A multinational survey

Background: This study aims to identify the preferred sources for acquiring knowledge about COVID-19 and to evaluate basic knowledge on critical scientific literature appraisal in students from medical schools located in Spanish speaking countries in Latin America. 

Methods: We designed an online survey of 15 closed-ended questions related to demographics, preferred resources for COVID-19 training, and items to assess critical appraisal skills. A snowball method was used for sampling. We conducted a descriptive analysis and Chi-squared tests to compare the proportion of correct identification of the concept of a preprint and a predatory journal when considering a) self-perceived level of knowledge, b) public vs private school, c) inclusion of a scientific literature appraisal subject in the curriculum, and d) progress in medical school.

Results: Our sample included 770 valid responses, out of which most of the participants included were from Mexico (n=283, 36.8%) and Ecuador (n=229, 29.7%). Participants preferred using evidence-based clinical resources (EBCRs) to learn more about COVID-19 (n=182, 23.6%). The preferred study design was case report/series (n=218, 28.1%). We found that only 265 participants correctly identified the concept of a preprint (34.4%), while 243 students (31.6%) correctly identified the characteristics of a predatory journal. We found no significant differences in the proportion of correct answers regardless of the self-perceived level of knowledge, progress in medical school, or scientific literature critical appraisal classes.

Conclusion: This study is novel in its approach of identifying sources of knowledge used by Latin American medical students and provides insights into the need to reinforce training in critical appraisal of scientific literature during medical school.

Segmenting clinicians’ usage patterns of a digital health tool in limited resource settings: methodology and initial results (Preprint)

Background

Evidence-based digital health tools allow clinicians to keep up with the expanding medical literature and provide safer and more accurate care. Understanding users’ online behavior in low-resource settings can inform programs that encourage the use of such tools. Our program collaborates with digital tool providers, including UpToDate, to facilitate free subscriptions for clinicians serving in low-resource settings globally.

Objective

We aimed to define segments of clinicians based on their usage patterns of UpToDate, describe the demographics of those segments, and relate the segments to self-reported professional climate measures.

Methods

We collected 12 months of clickstream data (a record of users’ clicks within the tool) as well as repeated surveys. We calculated the total number of sessions, time spent online, type of activity (navigating, reading, or account management), calendar period of use, percentage of days active online, and minutes of use per active day. We defined behavioral segments based on the distributions of these statistics and related them to survey data.

Results

We enrolled 1681 clinicians from 75 countries over a 9-week period. We based the following five behavioral segments on the length and intensity of use: short-term, light users (420/1681, 25%); short-term, heavy users (252/1681, 15%); long-term, heavy users (403/1681, 24%); long-term, light users (370/1681, 22%); and never-users (252/1681, 15%). Users spent a median of 5 hours using the tool over the year. On days when users logged on, they spent a median of 4.4 minutes online and an average of 71% of their time reading medical content as opposed to navigating or managing their account. Over half (773/1432, 54%) of the users actively used the tool for 48 weeks or more during the 52-week study period. The distribution of segments varied by age, with lighter and less use among those aged 35 years or older compared to that among younger users. The speciality of medicine had the heaviest use, and emergency medicine had the lightest use. Segments varied strongly by geographic region. As for professional climate, most respondents (1429/1681, 85%) reported that clinicians in their area would view the use of a online tool positively, and compared to those who reported other views, these respondents were less likely to be never-users (286/1681, 17% vs 387/1681, 23%) and more likely to be long-term users (655/1681, 39% vs 370/1681, 22%).

Conclusions

We believe that these behavioral segments can help inform the implementation of digital health tools, identify users who may need assistance, tailor training and messaging for users, and support research on digital health efforts. Methods for combining clickstream data with demographic and survey data have the potential to inform global health implementation. Our forthcoming analysis will use these methods to better elucidate what drives digital health tool use.

Association of Use of the NEJM Knowledge+ Product and Performance on the ABIM IM-MOC Exam

Phenomenon: Internal medicine physicians in the United States must pass the American Board of Internal Medicine Internal Medicine Maintenance of Certification (ABIM IM-MOC) examination as part of their ABIM IM-MOC requirements. Many of these physicians use an examination product to help them prepare, such as e-Learning products, including the ACP's MKSAP, UpToDate, and NEJM Knowledge+, yet their effectiveness remains largely unstudied. Approach: We compared ABIM IM-MOC examination performance among 177 physicians who attempted an ABIM IM-MOC examination between 2014-2017 and completed at least 75% of the NEJM Knowledge+ product prior to the ABIM IM-MOC examination and 177 very similar matched control physicians who did not use NEJM Knowledge+. Our measures of ABIM IM-MOC exam performance for NEJM Knowledge+ users were based on the results of the first attempt immediately following the NEJM Knowledge+ use and for non-users were based on the applicable matched examination performance. The three dichotomous examination performance outcomes measured on the first attempt at the ABIM IM-MOC examination included pass rate, scoring in the upper quartile, and scoring in the lower quartile. Findings: Use of NEJM Knowledge+ was associated with a regression adjusted 10.6% (5.37% to 15.8%) greater likelihood of passing the MOC examination (p < .001), 10.7% (2.61% to 18.7%) greater likelihood of having an examination score in the top quartile (p = .009), and -10.8% (-16.8% to -4.86%) lower likelihood of being in the bottom quartile of the MOC examination (p < .001) as compared to similar physicians who did not use NEJM Knowledge+. Insight: Physicians who used NEJM Knowledge+ had better ABIM IM-MOC exam performance. Further research is needed to determine what aspects of e-Learning products best prepare physicians for MOC examinations.

Definitions of Physician Certification Used in the North American Literature: A Scoping Review

INTRODUCTION

The authors sought to identify how physician specialty certification is defined in the North American literature.

METHODS

A rigorous, established six-stage scoping review framework was used to identify the North American certification literature published between January 2006 and May 2016 relating to physician specialty certification. Data were abstracted using a charting form developed by the study team. Quantitative summary data and qualitative thematic analysis of the purpose of certification were derived from the extracted data.

RESULTS

A two stage screening process identified 88 articles that met predefined criteria. Only 14 of the 88 articles (16%) contained a referenced purpose of certification. Eighteen definitions were identified from these articles. Definitional concepts included lifelong learning and continuous professional development, assessment of competence and performance, performance improvement, public accountability, and professional standing.

DISCUSSION

Most articles identified in this scoping review did not define certification or describe its purpose or intent. Future studies should provide a definition of certification to further scholarly examination of its intent and effects and inform its further evolution.

Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis

Background

Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use.

Objective

The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes.

Methods

We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies.

Results

Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I²>50%) across studies. When compared with other resources—7 studies, not amenable to meta-analytic pooling—the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource.

Conclusions

Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.

Electronic Knowledge Resources and Point-of-Care Learning: A Scoping Review

PURPOSE

The authors sought to summarize quantitative and qualitative research addressing electronic knowledge resources and point-of-care learning in a scoping review.

METHOD

The authors searched MEDLINE, Embase, PsycINFO, and the Cochrane Database for studies addressing electronic knowledge resources and point-of-care learning. They iteratively revised inclusion criteria and operational definitions of study features and research themes of interest. Two reviewers independently performed each phase of study selection and data extraction.

RESULTS

Of 10,811 studies identified, 305 were included and reviewed. Most studies (225; 74%) included physicians or medical students. The most frequently mentioned electronic resources were UpToDate (88; 29%), Micromedex (59; 19%), Epocrates (50; 16%), WebMD (46; 15%), MD Consult (32; 10%), and LexiComp (31; 10%). Eight studies (3%) evaluated electronic resources or point-of-care learning using outcomes of patient effects, and 36 studies (12%) reported objectively measured clinician behaviors. Twenty-five studies (8%) examined the clinical or educational impact of electronic knowledge resource use on patient care or clinician knowledge, 124 (41%) compared use rates of various knowledge resources, 69 (23%) examined the quality of knowledge resource content, and 115 (38%) explored the process of point-of-care learning. Two conceptual clarifications were identified, distinguishing the impact on clinical or educational outcomes versus the impact on test setting decision support, and the quality of information content versus the correctness of information obtained by a clinician-user.

CONCLUSIONS

Research on electronic knowledge resources is dominated by studies involving physicians and evaluating use rates. Studies involving nonphysician users, and evaluating resource impact and implementation, are needed.

Information needs of generalists and specialists using online best-practice algorithms to answer clinical questions

Objective

To better understand clinician information needs and learning opportunities by exploring the use of best-practice algorithms across different training levels and specialties.

Methods

We developed interactive online algorithms (care process models [CPMs]) that integrate current guidelines, recent evidence, and local expertise to represent cross-disciplinary best practices for managing clinical problems. We reviewed CPM usage logs from January 2014 to June 2015 and compared usage across specialty and provider type.

Results

During the study period, 4009 clinicians (2014 physicians in practice, 1117 resident physicians, and 878 nurse practitioners/physician assistants [NP/PAs]) viewed 140 CPMs a total of 81 764 times. Usage varied from 1 to 809 views per person, and from 9 to 4615 views per CPM. Residents and NP/PAs viewed CPMs more often than practicing physicians. Among 2742 users with known specialties, generalists ( N  = 1397) used CPMs more often (mean 31.8, median 7 views) than specialists ( N  = 1345; mean 6.8, median 2; P  < .0001). The topics used by specialists largely aligned with topics within their specialties. The top 20% of available CPMs (28/140) collectively accounted for 61% of uses. In all, 2106 clinicians (52%) returned to the same CPM more than once (average 7.8 views per topic; median 4, maximum 195). Generalists revisited topics more often than specialists (mean 8.8 vs 5.1 views per topic; P  < .0001).

Conclusions

CPM usage varied widely across topics, specialties, and individual clinicians. Frequently viewed and recurrently viewed topics might warrant special attention. Specialists usually view topics within their specialty and may have unique information needs.

Effectiveness of a clinical knowledge support system for reducing diagnostic errors in outpatient care in Japan: A retrospective study

Clinical evidence has indicated the effectiveness of computer-based systems for preventing and reducing diagnostic errors. Our study aimed to evaluate the effectiveness of UpToDate, a computer-based clinical knowledge management system, for reducing diagnostic errors. We retrospectively identified 100 patients who visited an outpatient department in a community-based hospital from July 2014 to June 2015. Fifty patients (exposure group) were seen by UpToDate-equipped physicians and another 50 (control group) were seen by UpToDate-unequipped physicians. We extracted data on patient sex, age, primary diagnosis, and case difficulty that could potentially affect diagnostic outcomes. We compared the two groups regarding diagnostic error rate and performed logistic regression analysis to analyze the concurrent effects of various factors affecting diagnostic error. The diagnostic error rate was 2% in the exposure group, while the error rate was 24% in the control group. Multivariate logistic regression analysis showed that error rate reduction was significantly associated with exposure to UpToDate with an odds ratio of 15.21 (95% CI 1.86-124.36). Our results demonstrated the effectiveness of UpToDate for the prevention and reduction of diagnostic error.

Optimization of infobutton design and Implementation: A systematic review

OBJECTIVE

Infobuttons are clinical decision tools embedded in the electronic health record that attempt to link clinical data with context sensitive knowledge resources. We systematically reviewed technical approaches that contribute to improved infobutton design, implementation and functionality.

METHODS

We searched databases including MEDLINE, EMBASE, and the Cochrane Library database from inception to March 1, 2016 for studies describing the use of infobuttons. We selected full review comparative studies, usability studies, and qualitative studies examining infobutton design and implementation. We abstracted usability measures such as user satisfaction, impact, and efficiency, as well as prediction accuracy of infobutton content retrieval algorithms and infobutton adoption/interoperability.

RESULTS

We found 82 original research studies on infobuttons. Twelve studies met criteria for detailed abstraction. These studies investigated infobutton interoperability (1 study); tools to help tailor infobutton functionality (1 study); interventions to improve user experience (7 studies); and interventions to improve content retrieval by improving prediction of relevant knowledge resources and information needs (3 studies). In-depth interviews with implementers showed the Health Level Seven (HL7) Infobutton standard to be simple and easy to implement. A usability study demonstrated the feasibility of a tool to help medical librarians tailor infobutton functionality. User experience studies showed that access to resources with which users are familiar increased user satisfaction ratings; and that links to specific subsections of drug monographs increased information seeking efficiency. However, none of the user experience improvements led to increased usage uptake. Recommender systems based on machine learning algorithms outperformed hand-crafted rules in the prediction of relevant resources and clinicians' information needs in a laboratory setting, but no studies were found using these techniques in clinical settings. Improved content indexing in one study led to improved content retrieval across three health care organizations.

CONCLUSION

Best practice technical approaches to ensure optimal infobutton functionality, design and implementation remain understudied. The HL7 Infobutton standard has supported wide adoption of infobutton functionality among clinical information systems and knowledge resources. Limited evidence supports infobutton enhancements such as links to specific subtopics, configuration of optimal resources for specific tasks and users, and improved indexing and content coverage. Further research is needed to investigate user experience improvements to increase infobutton use and effectiveness.

Evidence-based medicine for all: what we can learn from a programme providing free access to an online clinical resource to health workers in resource-limited settings

The rapidly changing landscape of medical knowledge and guidelines requires health professionals to have immediate access to current, reliable clinical resources. Access to evidence is instrumental in reducing diagnostic errors and generating better health outcomes. UpToDate, a leading evidence-based clinical resource is used extensively in the USA and other regions of the world and has been linked to lower mortality and length of stay in US hospitals. In 2009, the Global Health Delivery Project collaborated with UpToDate to provide free subscriptions to qualifying health workers in resource-limited settings. We evaluated the provision of UpToDate access to health workers by analysing their usage patterns. Since 2009, ∼2000 individual physicians and healthcare institutions from 116 countries have received free access to UpToDate through our programme. During 2013-2014, users logged into UpToDate ∼150 000 times; 61% of users logged in at least weekly; users in Africa were responsible for 54% of the total usage. Search patterns reflected local epidemiology with 'clinical manifestations of malaria' as the top search in Africa, and 'management of hepatitis B' as the top search in Asia. Our programme demonstrates that there are barriers to evidence-based clinical knowledge in resource-limited settings we can help remove. Some assumed barriers to its expansion (poor internet connectivity, lack of training and infrastructure) might pose less of a burden than subscription fees.

Context-sensitive decision support (infobuttons) in electronic health records: a systematic review

OBJECTIVE

Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes.

METHODS

We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information.

RESULTS

Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes.

CONCLUSIONS

Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.

Considering Point-of-Care Electronic Medical Resources in Lieu of Traditional Textbooks for Medical Education

Selecting resources to support didactic courses is a critical decision, and the advantages and disadvantages must be carefully considered. During clinical rotations, students not only need to possess strong background knowledge but also are expected to be proficient with the same evidence-based POC resources used by clinicians. Students place high value on “real world” learning and therefore may place more value on POC resources that they know practicing clinicians use as compared with medical textbooks. The condensed nature of PA education requires students to develop background knowledge and information literacy skills over a short period. One way to build that knowledge and those skills simultaneously is to use POC resources in lieu of traditional medical textbooks during didactic training. Electronic POC resources offer several advantages over traditional textbooks and should be considered as viable options in PA education.

An automated clinical alert system for newly-diagnosed atrial fibrillation

OBJECTIVE

Clinical decision support systems that notify providers of abnormal test results have produced mixed results. We sought to develop, implement, and evaluate the impact of a computer-based clinical alert system intended to improve atrial fibrillation stroke prophylaxis, and identify reasons providers do not implement a guideline-concordant response.

MATERIALS AND METHODS

We conducted a cohort study with historical controls among patients at a tertiary care hospital. We developed a decision rule to identify newly-diagnosed atrial fibrillation, automatically notify providers, and direct them to online evidence-based management guidelines. We tracked all notifications from December 2009 to February 2010 (notification period) and applied the same decision rule to all patients from December 2008 to February 2009 (control period). Primary outcomes were accuracy of notification (confirmed through chart review) and prescription of warfarin within 30 days.

RESULTS

During the notification period 604 notifications were triggered, of which 268 (44%) were confirmed as newly-diagnosed atrial fibrillation. The notifications not confirmed as newly-diagnosed involved patients with no recent electrocardiogram at our institution. Thirty-four of 125 high-risk patients (27%) received warfarin in the notification period, compared with 34 of 94 (36%) in the control period (odds ratio, 0.66 [95% CI, 0.37-1.17]; p = 0.16). Common reasons to not prescribe warfarin (identified from chart review of 151 patients) included upcoming surgical procedure, choice to use aspirin, and discrepancy between clinical notes and the medication record.

CONCLUSIONS

An automated system to identify newly-diagnosed atrial fibrillation, notify providers, and encourage access to management guidelines did not change provider behaviors.

A comprehensive information technology system to support physician learning at the point of care

MayoExpert is a multifaceted information system integrated with the electronic medical record (EMR) across Mayo Clinic's multisite health system. It was developed as a technology-based solution to manage information, standardize clinical practice, and promote and document learning in clinical contexts. Features include urgent test result notifications; models illustrating expert-approved care processes; concise, expert-approved answers to frequently asked questions (FAQs); a directory of topic-specific experts; and a portfolio for provider licensure and credentialing. The authors evaluate MayoExpert's reach, effectiveness, adoption, implementation, and maintenance. Evaluation data sources included usage statistics, user surveys, and pilot studies.As of October 2013, MayoExpert was available at 94 clinical sites in 12 states and contained 1,368 clinical topics, answers to 7,640 FAQs, and 92 care process models. In 2012, MayoExpert was accessed at least once by 2,578/3,643 (71%) staff physicians, 900/1,374 (66%) midlevel providers, and 1,728/2,291 (75%) residents and fellows. In a 2013 survey of MayoExpert users with 536 respondents, all features were highly rated (≥67% favorable). More providers reported using MayoExpert to answer questions before/after than during patient visits (68% versus 36%). During November 2012 to April 2013, MayoExpert sent 1,660 notifications of new-onset atrial fibrillation and 1,590 notifications of prolonged QT. MayoExpert has become part of routine clinical and educational operations, and its care process models now define Mayo Clinic best practices. MayoExpert's infrastructure and content will continue to expand with improved templates and content organization, new care process models, additional notifications, better EMR integration, and improved support for credentialing activities.

Recent trends in internal medicine education: a brief update

This perspective attempts to bring graduate medical offices, residency programs and medical students interested in categorical internal medicine (CIM) a brief update on the American Board of Internal Medicine (ABIM), Accreditation Council for Graduate Medical Education (ACGME) and the National Registry for Medical Programs (NRMP) changes for the past 3-5 years in the United States (US). The US model for certification and recertification may serve as a homogenous example for other countries. This model will be described so that there is an understanding of the importance of such changes in the American system and its effect on resident education. This is critical knowledge for both teachers and learners in internal medicine in preparation for a lifetime career and requirements for certification/credentialing for both programs and their residents/fellows. Data from the review indicate a small increase in the number of applicants but a concordant decrease in ABIM initial certification exams. Programs should well be aware of the new focus on outcomes via the Next Accreditation System (NAS) being put forth by the ACGME.

Assessing clinical reasoning: moving from in vitro to in vivo

The last century saw dramatic changes in clinical practice and medical education and the concomitant rise in high-stakes, psychometrically-based examinations of medical knowledge. Higher scores on these high-stakes “in-vitro” examinations are modestly associated with better performance in clinical practice and provide a meaningful degree of assurance to the public about physicians’ competency in medical knowledge. However, results on such examinations explain only a small fraction of the wide variation currently seen in clinical practice and diagnostic errors remain a serious and vexing problem for patients and the healthcare system despite decades of high-stakes examinations. In this commentary we explore some of the limitations of high-stakes examinations in assessing clinical reasoning and propose utilizing situated cognition theory to guide research and development of innovative modes of ”in-vivo” assessments that can be used in longitudinally and continuously in clinical practice.

Features of effective medical knowledge resources to support point of care learning: a focus group study

OBJECTIVE

Health care professionals access various information sources to quickly answer questions that arise in clinical practice. The features that favorably influence the selection and use of knowledge resources remain unclear. We sought to better understand how clinicians select among the various knowledge resources available to them, and from this to derive a model for an effective knowledge resource.

METHODS

We conducted 11 focus groups at an academic medical center and outlying community sites. We included a purposive sample of 50 primary care and subspecialist internal medicine and family medicine physicians. We transcribed focus group discussions and analyzed these using a constant comparative approach to inductively identify features that influence the selection of knowledge resources.

RESULTS

We identified nine features that influence users' selection of knowledge resources, namely efficiency (with sub-features of comprehensiveness, searchability, and brevity), integration with clinical workflow, credibility, user familiarity, capacity to identify a human expert, reflection of local care processes, optimization for the clinical question (e.g., diagnosis, treatment options, drug side effect), currency, and ability to support patient education. No single existing resource exemplifies all of these features.

CONCLUSION

The influential features identified in this study will inform the development of knowledge resources, and could serve as a framework for future research in this field.

How is clinical reasoning developed, maintained, and objectively assessed? Views from expert internists and internal medicine interns

INTRODUCTION

There is limited understanding of how clinical reasoning is developed, maintained, and objectively assessed. Using the theoretical lens of deliberate practice, we explored physicians' views on clinical reasoning. We compared responses from internists (faculty) and internal medicine interns, to identify potential qualitative and/or quantitative differences in how clinical reasoning is developed and maintained.

METHODS

Participants' free-text comments regarding how clinical reasoning is developed, maintained, and objectively assessed were analyzed. Three investigators coded responses using a constant-comparative, grounded theory approach. We also compared the frequencies of each theme between the 2 groups.

RESULTS

Twenty-two faculty and 17 interns participated in this study. Faculty and intern themes for how clinical reasoning is developed, maintained, and objectively assessed were similar, but quantitative and qualitative differences emerged. Interrater reliability of themes was high (overall kappa: 0.92; range: 0.88-0.98). Only experts (faculty) mentioned the value of teaching for development and maintenance of clinical reasoning. Interns focused on knowledge acquisition activities and use of online resources. Experts and intern participants both struggled with how to best measure clinical reasoning; direct observation was rarely mentioned as a strategy.

DISCUSSION

Consistent with our theoretical expectations, we found quantitative and qualitative differences in participants' responses, which have implications for teaching and assessment of clinical reasoning. By capturing the types of activities and their relative frequencies within and between these groups, this work adds to the deliberate practice literature.

Specialty board certification in the United States: issues and evidence

BACKGROUND

The American Board of Medical Specialties (ABMS) certification and maintenance of certification (MOC) programs strive to provide the public with guidance about a physician's competence. This study summarizes the literature on the effectiveness of these programs.

METHOD

A literature search was conducted for studies published between 1986 and April 2013 and limited to ABMS certification. A modified version of Kirkpatrick's 4 levels of program evaluation included the reaction of stakeholders to certification, the extent to which physicians are encouraged to improve, the relationship between performance in the programs and nonclinical external measures of physician competence, and the relationship of performance in the programs with clinical quality measures.

RESULTS

Patients' and hospitals' value of board certification and physician participation in MOC are high. Physicians are conflicted as to whether the effort involved is worth its value. Self-reported evidence shows improvement in knowledge, practice infrastructure, communication with patients and peers, and clinical care. Certification performance is generally related to nonclinical external measures such as types of training, practice characteristics, demographics, and disciplinary actions. In general, physicians who are board certified provide better patient care, albeit the results have modest effect sizes and are not unequivocal.

CONCLUSIONS

Certification boards should continuously try to improve their programs in response to feedback from stakeholders, changes in the way physicians practice, as well as the growth in the fields of measurement and technology. Keeping pace with these changes in a responsible and evidence-based way is important.

Assessment in the context of licensure and certification

Over the past 25 years, three major forces have had a significant influence on licensure and certification: the shift in focus from educational process to educational outcomes, the increasing recognition of the need for learning and assessment throughout a physician's career, and the changes in technology and psychometrics that have opened new vistas for assessment. These forces have led to significant changes in assessment for licensure and certification. To respond to these forces, licensure and certification programs have improved the ways in which their examinations are constructed, scored, and delivered. In particular, we note the introduction of adaptive testing; automated item creation, scoring, and test assembly; assessment engineering; and data forensics. Licensure and certification programs have also expanded their repertoire of assessments with the rapid development and adoption of simulation and workplace-based assessment. Finally, they have invested in research intended to validate their programs in four ways: (a) the acceptability of the program to stakeholders, (b) the extent to which stakeholders are encouraged to learn and improve, (c) the extent to which there is a relationship between performance in the programs and external measures, and (d) the extent to which there is a relationship between performance as measured by the assessment and performance in practice. Over the past 25 years, changes in licensure and certification have been driven by the educational outcomes movement, the need for lifelong learning, and advances in technology and psychometrics. Over the next 25 years, we expect these forces to continue to exert pressure for change which will lead to additional improvement and expansion in examination processes, methods of assessment, and validation research.