1
|
Calderon Y, Sandigan G, Tan-Lim CSC, De Mesa RYH, Fabian NMC, Rey MP, Sanchez JT, Dans LF, Galingana CLT, Bernal-Sundiang N, Casile RU, Aquino MRN, Poblete KE, Lopez JFE, Zabala H, Dans AL. Feasibility, acceptability and impact of a clinical decision support tool among primary care providers in an urban, rural and remote site in the Philippines. BMJ Open Qual 2024; 13:e002526. [PMID: 38423587 PMCID: PMC10910488 DOI: 10.1136/bmjoq-2023-002526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2023] [Accepted: 02/22/2024] [Indexed: 03/02/2024] Open
Abstract
INTRODUCTION Strengthening primary care helps address health inequities that continue to persist in the Philippines. The Philippine Primary Care Studies pilot-tested interventions to improve the primary care system. One intervention was the provision of a free subscription to an electronic decision support application called UpToDate (UTD) for primary care providers (PCPs), including doctors, nurses, midwives and community health workers (CHWs). This study aimed to (1) assess the feasibility of using UTD as information source for PCPs in urban, rural and remote settings, (2) determine the acceptability of UTD as an information source for PCPs and (3) examine the impact of UTD access on PCP clinical decision-making. METHODS Four focus group discussions (FGDs) and two key informant interviews (KII) were conducted to gather insights from 30 PCPs. Thematic analysis through coding in NVivo V.12 was done using the technology acceptance model (TAM) as a guiding framework. RESULTS All PCPs had positive feedback regarding UTD use because of its comprehensiveness, accessibility, mobility and general design. The participants relayed UTD's benefit for point-of-contact use, capacity-building and continuing professional development. PCPs across the three sites, including CHWs with no formal medical education, were able to provide evidence-based medical advice to patients through UTD. However, external factors in these settings impeded the full integration of UTD in the PCPs' workflow, including poor internet access, unstable sources of electricity, lack of compatible mobile devices and the need for translation to the local language. CONCLUSION UTD was a feasible and acceptable clinical decision support tool for the PCPs. Factors affecting the feasibility of using UTD include technological and environmental factors (ie, internet access and the lack of translation to the local language), as well as the organisational structure of the primary care facility which determines the roles of the PCPs. Despite the difference in roles and responsibilities of the PCPs, UTD positively impacted decision-making and patient education for all PCPs through its use as a point-of-contact tool and a tool for capacity-building.
Collapse
Affiliation(s)
- Ysabela Calderon
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Gillian Sandigan
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Carol Stephanie C Tan-Lim
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
- Department of Clinical Epidemiology, University of the Philippines Manila, Manila, Philippines
| | - Regine Ynez H De Mesa
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Noleen Marie C Fabian
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Mia P Rey
- Department of Accounting and Finance, Cesar E.A. Virata School of Business, University of the Philippines Diliman, Quezon City, Philippines
| | - Josephine T Sanchez
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Leonila F Dans
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
- Department of Clinical Epidemiology, University of the Philippines Manila, Manila, Philippines
| | - Cara Lois T Galingana
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Nannette Bernal-Sundiang
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Ray U Casile
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Maria Rhodora N Aquino
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Karl Engelene Poblete
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Johanna Faye E Lopez
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Herbert Zabala
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
| | - Antonio L Dans
- Center for Integrative and Development Studies, University of the Philippines Diliman, Quezon City, Metro Manila, Philippines
- National Institutes of Health, University of the Philippines Manila, Manila, Metro Manila, Philippines
| |
Collapse
|
2
|
Nicolalde B, Añazco D, Jaramillo-Cartwright MJ, Salinas I, Pacheco-Carrillo A, Hernández-Chávez S, Moyano G, Teran E. Scientific literacy and preferred resources used by Latin American medical students during the COVID-19 pandemic: A multinational survey. F1000Res 2022; 11:341. [PMID: 35919099 PMCID: PMC9296994 DOI: 10.12688/f1000research.109398.2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/28/2022] [Indexed: 12/01/2022] Open
Abstract
Background: This study aims to identify the preferred sources for acquiring knowledge about COVID-19 and to evaluate basic knowledge on critical scientific literature appraisal in students from medical schools located in Spanish speaking countries in Latin America. Methods: We designed an online survey of 15 closed-ended questions related to demographics, preferred resources for COVID-19 training, and items to assess critical appraisal skills. A snowball method was used for sampling. We conducted a descriptive analysis and Chi-squared tests to compare the proportion of correct identification of the concept of a preprint and a predatory journal when considering a) self-perceived level of knowledge, b) public vs private school, c) inclusion of a scientific literature appraisal subject in the curriculum, and d) progress in medical school. Results: Our sample included 770 valid responses, out of which most of the participants included were from Mexico (n=283, 36.8%) and Ecuador (n=229, 29.7%). Participants preferred using evidence-based clinical resources (EBCRs) to learn more about COVID-19 (n=182, 23.6%). The preferred study design was case report/series (n=218, 28.1%). We found that only 265 participants correctly identified the concept of a preprint (34.4%), while 243 students (31.6%) correctly identified the characteristics of a predatory journal. We found no significant differences in the proportion of correct answers regardless of the self-perceived level of knowledge, progress in medical school, or scientific literature critical appraisal classes. Conclusion: This study is novel in its approach of identifying sources of knowledge used by Latin American medical students and provides insights into the need to reinforce training in critical appraisal of scientific literature during medical school.
Collapse
Affiliation(s)
- Bryan Nicolalde
- Colegio de Ciencias de la Salud, Universidad San Francisco de Quito, Quito, Ecuador
| | - Diego Añazco
- Colegio de Ciencias de la Salud, Universidad San Francisco de Quito, Quito, Ecuador
| | | | - Ivonne Salinas
- Colegio de Ciencias de la Salud, Universidad San Francisco de Quito, Quito, Ecuador
| | | | | | - Gimena Moyano
- Fundación H. Barceló, Instituto Universitario de Ciencias de la Salud, La Rioja, Argentina
| | - Enrique Teran
- Colegio de Ciencias de la Salud, Universidad San Francisco de Quito, Quito, Ecuador
| |
Collapse
|
3
|
Miller K, Rosenberg J, Pickard O, Hawrusik R, Karlage A, Weintraub R. Segmenting clinicians’ usage patterns of a digital health tool in limited resource settings: methodology and initial results (Preprint). JMIR Form Res 2021; 6:e30320. [PMID: 35532985 PMCID: PMC9127647 DOI: 10.2196/30320] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2021] [Revised: 02/14/2022] [Accepted: 02/25/2022] [Indexed: 11/13/2022] Open
Abstract
Background Evidence-based digital health tools allow clinicians to keep up with the expanding medical literature and provide safer and more accurate care. Understanding users’ online behavior in low-resource settings can inform programs that encourage the use of such tools. Our program collaborates with digital tool providers, including UpToDate, to facilitate free subscriptions for clinicians serving in low-resource settings globally. Objective We aimed to define segments of clinicians based on their usage patterns of UpToDate, describe the demographics of those segments, and relate the segments to self-reported professional climate measures. Methods We collected 12 months of clickstream data (a record of users’ clicks within the tool) as well as repeated surveys. We calculated the total number of sessions, time spent online, type of activity (navigating, reading, or account management), calendar period of use, percentage of days active online, and minutes of use per active day. We defined behavioral segments based on the distributions of these statistics and related them to survey data. Results We enrolled 1681 clinicians from 75 countries over a 9-week period. We based the following five behavioral segments on the length and intensity of use: short-term, light users (420/1681, 25%); short-term, heavy users (252/1681, 15%); long-term, heavy users (403/1681, 24%); long-term, light users (370/1681, 22%); and never-users (252/1681, 15%). Users spent a median of 5 hours using the tool over the year. On days when users logged on, they spent a median of 4.4 minutes online and an average of 71% of their time reading medical content as opposed to navigating or managing their account. Over half (773/1432, 54%) of the users actively used the tool for 48 weeks or more during the 52-week study period. The distribution of segments varied by age, with lighter and less use among those aged 35 years or older compared to that among younger users. The speciality of medicine had the heaviest use, and emergency medicine had the lightest use. Segments varied strongly by geographic region. As for professional climate, most respondents (1429/1681, 85%) reported that clinicians in their area would view the use of a online tool positively, and compared to those who reported other views, these respondents were less likely to be never-users (286/1681, 17% vs 387/1681, 23%) and more likely to be long-term users (655/1681, 39% vs 370/1681, 22%). Conclusions We believe that these behavioral segments can help inform the implementation of digital health tools, identify users who may need assistance, tailor training and messaging for users, and support research on digital health efforts. Methods for combining clickstream data with demographic and survey data have the potential to inform global health implementation. Our forthcoming analysis will use these methods to better elucidate what drives digital health tool use.
Collapse
Affiliation(s)
- Kate Miller
- Better Evidence, Ariadne Labs, Boston, MA, United States
- Department of Health Policy and Management, Harvard TH Chan School of Public Health, Boston, MA, United States
| | - Julie Rosenberg
- Better Evidence, Ariadne Labs, Boston, MA, United States
- Division of Global Health Equity, Brigham and Women's Hospital, Boston, MA, United States
| | - Olivia Pickard
- Better Evidence, Ariadne Labs, Boston, MA, United States
| | | | - Ami Karlage
- Better Evidence, Ariadne Labs, Boston, MA, United States
| | | |
Collapse
|
4
|
Healy MG, Gray BM, Duhigg LM, O'Rourke M, Marchwinski L, Elmore M, Phitayakorn R. Association of Use of the NEJM Knowledge+ Product and Performance on the ABIM IM-MOC Exam. TEACHING AND LEARNING IN MEDICINE 2021; 33:21-27. [PMID: 32928000 DOI: 10.1080/10401334.2020.1811095] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Phenomenon: Internal medicine physicians in the United States must pass the American Board of Internal Medicine Internal Medicine Maintenance of Certification (ABIM IM-MOC) examination as part of their ABIM IM-MOC requirements. Many of these physicians use an examination product to help them prepare, such as e-Learning products, including the ACP's MKSAP, UpToDate, and NEJM Knowledge+, yet their effectiveness remains largely unstudied. Approach: We compared ABIM IM-MOC examination performance among 177 physicians who attempted an ABIM IM-MOC examination between 2014-2017 and completed at least 75% of the NEJM Knowledge+ product prior to the ABIM IM-MOC examination and 177 very similar matched control physicians who did not use NEJM Knowledge+. Our measures of ABIM IM-MOC exam performance for NEJM Knowledge+ users were based on the results of the first attempt immediately following the NEJM Knowledge+ use and for non-users were based on the applicable matched examination performance. The three dichotomous examination performance outcomes measured on the first attempt at the ABIM IM-MOC examination included pass rate, scoring in the upper quartile, and scoring in the lower quartile. Findings: Use of NEJM Knowledge+ was associated with a regression adjusted 10.6% (5.37% to 15.8%) greater likelihood of passing the MOC examination (p < .001), 10.7% (2.61% to 18.7%) greater likelihood of having an examination score in the top quartile (p = .009), and -10.8% (-16.8% to -4.86%) lower likelihood of being in the bottom quartile of the MOC examination (p < .001) as compared to similar physicians who did not use NEJM Knowledge+. Insight: Physicians who used NEJM Knowledge+ had better ABIM IM-MOC exam performance. Further research is needed to determine what aspects of e-Learning products best prepare physicians for MOC examinations.
Collapse
Affiliation(s)
- Michael G Healy
- Department of Surgery, Harvard Medical School, Massachusetts General Hospital, Boston, Massachusetts, USA
- NEJM Group, Waltham, Massachusetts, USA
| | - Bradley M Gray
- American Board of Internal Medicine, Philadelphia, Pennsylvania, USA
| | - Lauren M Duhigg
- American Board of Internal Medicine, Philadelphia, Pennsylvania, USA
| | | | | | | | - Roy Phitayakorn
- Department of Surgery, Harvard Medical School, Massachusetts General Hospital, Boston, Massachusetts, USA
- NEJM Group, Waltham, Massachusetts, USA
| |
Collapse
|
5
|
Price D, Campbell C, Van Hoof TJ, ElChamaa R, Jeong D, Chappell K, Moore D, Olson C, Danilovich N, Kitto S. Definitions of Physician Certification Used in the North American Literature: A Scoping Review. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2020; 40:147-157. [PMID: 32898116 DOI: 10.1097/ceh.0000000000000312] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION The authors sought to identify how physician specialty certification is defined in the North American literature. METHODS A rigorous, established six-stage scoping review framework was used to identify the North American certification literature published between January 2006 and May 2016 relating to physician specialty certification. Data were abstracted using a charting form developed by the study team. Quantitative summary data and qualitative thematic analysis of the purpose of certification were derived from the extracted data. RESULTS A two stage screening process identified 88 articles that met predefined criteria. Only 14 of the 88 articles (16%) contained a referenced purpose of certification. Eighteen definitions were identified from these articles. Definitional concepts included lifelong learning and continuous professional development, assessment of competence and performance, performance improvement, public accountability, and professional standing. DISCUSSION Most articles identified in this scoping review did not define certification or describe its purpose or intent. Future studies should provide a definition of certification to further scholarly examination of its intent and effects and inform its further evolution.
Collapse
Affiliation(s)
- David Price
- Dr. Price: The University of CO School of Medicine and Senior Advisor to the President, American Board of Family Medicine, Lexington, KY; Dr. Campbell: Associate Professor at the Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; Dr. Thomas Van Hoof: Associate Professor at the School of Nursing and School of Medicine, University of Connecticut, Mansfield, CT; Dr. Chappell: Senior Vice President of the Accreditation Program and Institute for Credentialing Research, American Nurses Credentialing Center, Silver Spring, MD; Dr. Moore: Director at the Division of Continuing Medical Education and Director of Evaluation and Education, Office of Graduate Medical Education, Vanderbilt University, Nashville, TN; and Professor of Medical Education and Administration, Vanderbilt University, Nashville, TN; Dr. Olson: Assistant Professor at the Geisel School of Medicine, Dartmouth College, Hanover, NH; Ms. ElChamaa: Research Associate at the Department of Innovation in Medical Education, and the Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa, Ottawa, Canada; Ms. Jeong: Research Associate at the Department of Innovation in Medical Education, and the Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa, Ottawa, Canada; Dr. Danilovich: Research Associate at the Department of Innovation in Medical Education, and the Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa, Ottawa, Canada; and Dr. Kitto: Professor at the Department of Innovation in Medical Education and the Faculty of Education, and Director of Research at the Office of Continuing Professional Development, Faculty of Medicine, University of Ottawa; and Assistant Professor at the Department of Surgery, University of Toronto, Canada
| | | | | | | | | | | | | | | | | | | |
Collapse
|
6
|
Maggio LA, Aakre CA, Del Fiol G, Shellum J, Cook DA. Impact of Clinicians' Use of Electronic Knowledge Resources on Clinical and Learning Outcomes: Systematic Review and Meta-Analysis. J Med Internet Res 2019; 21:e13315. [PMID: 31359865 PMCID: PMC6690166 DOI: 10.2196/13315] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2019] [Revised: 05/12/2019] [Accepted: 06/18/2019] [Indexed: 12/19/2022] Open
Abstract
Background Clinicians use electronic knowledge resources, such as Micromedex, UpToDate, and Wikipedia, to deliver evidence-based care and engage in point-of-care learning. Despite this use in clinical practice, their impact on patient care and learning outcomes is incompletely understood. A comprehensive synthesis of available evidence regarding the effectiveness of electronic knowledge resources would guide clinicians, health care system administrators, medical educators, and informaticians in making evidence-based decisions about their purchase, implementation, and use. Objective The aim of this review is to quantify the impact of electronic knowledge resources on clinical and learning outcomes. Methods We searched MEDLINE, Embase, PsycINFO, and the Cochrane Library for articles published from 1991 to 2017. Two authors independently screened studies for inclusion and extracted outcomes related to knowledge, skills, attitudes, behaviors, patient effects, and cost. We used random-effects meta-analysis to pool standardized mean differences (SMDs) across studies. Results Of 10,811 studies screened, we identified 25 eligible studies published between 2003 and 2016. A total of 5 studies were randomized trials, 22 involved physicians in practice or training, and 10 reported potential conflicts of interest. A total of 15 studies compared electronic knowledge resources with no intervention. Of these, 7 reported clinician behaviors, with a pooled SMD of 0.47 (95% CI 0.27 to 0.67; P<.001), and 8 reported objective patient effects with a pooled SMD of 0.19 (95% CI 0.07 to 0.32; P=.003). Heterogeneity was large (I2>50%) across studies. When compared with other resources—7 studies, not amenable to meta-analytic pooling—the use of electronic knowledge resources was associated with increased frequency of answering questions and perceived benefits on patient care, with variable impact on time to find an answer. A total of 2 studies compared different implementations of the same electronic knowledge resource. Conclusions Use of electronic knowledge resources is associated with a positive impact on clinician behaviors and patient effects. We found statistically significant associations between the use of electronic knowledge resources and improved clinician behaviors and patient effects. When compared with other resources, the use of electronic knowledge resources was associated with increased success in answering clinical questions, with variable impact on speed. Comparisons of different implementation strategies of the same electronic knowledge resource suggest that there are benefits from allowing clinicians to choose to access the resource, versus automated display of resource information, and from integrating patient-specific information. A total of 4 studies compared different commercial electronic knowledge resources, with variable results. Resource implementation strategies can significantly influence outcomes but few studies have examined such factors.
Collapse
Affiliation(s)
- Lauren A Maggio
- Department of Medicine, Uniformed Services University of the Health Sciences, Bethesda, MD, United States
| | - Christopher A Aakre
- Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN, United States
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States
| | - Jane Shellum
- Center for Translational Informatics and Knowledge Management, Mayo Clinic, Rochester, MN, United States
| | - David A Cook
- Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN, United States
| |
Collapse
|
7
|
Aakre CA, Pencille LJ, Sorensen KJ, Shellum JL, Del Fiol G, Maggio LA, Prokop LJ, Cook DA. Electronic Knowledge Resources and Point-of-Care Learning: A Scoping Review. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:S60-S67. [PMID: 30365431 DOI: 10.1097/acm.0000000000002375] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
PURPOSE The authors sought to summarize quantitative and qualitative research addressing electronic knowledge resources and point-of-care learning in a scoping review. METHOD The authors searched MEDLINE, Embase, PsycINFO, and the Cochrane Database for studies addressing electronic knowledge resources and point-of-care learning. They iteratively revised inclusion criteria and operational definitions of study features and research themes of interest. Two reviewers independently performed each phase of study selection and data extraction. RESULTS Of 10,811 studies identified, 305 were included and reviewed. Most studies (225; 74%) included physicians or medical students. The most frequently mentioned electronic resources were UpToDate (88; 29%), Micromedex (59; 19%), Epocrates (50; 16%), WebMD (46; 15%), MD Consult (32; 10%), and LexiComp (31; 10%). Eight studies (3%) evaluated electronic resources or point-of-care learning using outcomes of patient effects, and 36 studies (12%) reported objectively measured clinician behaviors. Twenty-five studies (8%) examined the clinical or educational impact of electronic knowledge resource use on patient care or clinician knowledge, 124 (41%) compared use rates of various knowledge resources, 69 (23%) examined the quality of knowledge resource content, and 115 (38%) explored the process of point-of-care learning. Two conceptual clarifications were identified, distinguishing the impact on clinical or educational outcomes versus the impact on test setting decision support, and the quality of information content versus the correctness of information obtained by a clinician-user. CONCLUSIONS Research on electronic knowledge resources is dominated by studies involving physicians and evaluating use rates. Studies involving nonphysician users, and evaluating resource impact and implementation, are needed.
Collapse
Affiliation(s)
- Christopher A Aakre
- C.A. Aakre is assistant professor of medicine and senior associate consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota. L.J. Pencille is program coordinator, Knowledge and Delivery Center, Center for Translational Informatics and Knowledge Management, Mayo Clinic, Rochester, Minnesota. K.J. Sorensen is assistant professor of medical education and unit head, Knowledge Management Technologies, Center for Translational Informatics and Knowledge Management, Mayo Clinic, Rochester, Minnesota. J.L. Shellum is section head, Knowledge and Delivery Center, Center for Translational Informatics and Knowledge Management, Mayo Clinic, Rochester, Minnesota. G. Del Fiol is assistant professor of biomedical informatics, University of Utah School of Medicine, Salt Lake City, Utah, and co-chair, Clinical Decision Support Work Group at Health Level Seven (HL7). L.A. Maggio is associate professor of medicine and associate director of technology and distributed learning, Department of Medicine, Uniformed Services University, Bethesda, Maryland. L.J. Prokop is reference librarian, Plummer Library, Mayo Clinic, Rochester, Minnesota. D.A. Cook is professor of medicine and medical education; researcher, Center for Translational Informatics and Knowledge Management; associate director, Office of Applied Scholarship and Education Science; and consultant, Division of General Internal Medicine, Mayo Clinic College of Medicine and Science, Rochester, Minnesota
| | | | | | | | | | | | | | | |
Collapse
|
8
|
Cook DA, Sorensen KJ, Linderbaum JA, Pencille LJ, Rhodes DJ. Information needs of generalists and specialists using online best-practice algorithms to answer clinical questions. J Am Med Inform Assoc 2018; 24:754-761. [PMID: 28339685 DOI: 10.1093/jamia/ocx002] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2016] [Accepted: 12/31/2016] [Indexed: 11/13/2022] Open
Abstract
Objective To better understand clinician information needs and learning opportunities by exploring the use of best-practice algorithms across different training levels and specialties. Methods We developed interactive online algorithms (care process models [CPMs]) that integrate current guidelines, recent evidence, and local expertise to represent cross-disciplinary best practices for managing clinical problems. We reviewed CPM usage logs from January 2014 to June 2015 and compared usage across specialty and provider type. Results During the study period, 4009 clinicians (2014 physicians in practice, 1117 resident physicians, and 878 nurse practitioners/physician assistants [NP/PAs]) viewed 140 CPMs a total of 81 764 times. Usage varied from 1 to 809 views per person, and from 9 to 4615 views per CPM. Residents and NP/PAs viewed CPMs more often than practicing physicians. Among 2742 users with known specialties, generalists ( N = 1397) used CPMs more often (mean 31.8, median 7 views) than specialists ( N = 1345; mean 6.8, median 2; P < .0001). The topics used by specialists largely aligned with topics within their specialties. The top 20% of available CPMs (28/140) collectively accounted for 61% of uses. In all, 2106 clinicians (52%) returned to the same CPM more than once (average 7.8 views per topic; median 4, maximum 195). Generalists revisited topics more often than specialists (mean 8.8 vs 5.1 views per topic; P < .0001). Conclusions CPM usage varied widely across topics, specialties, and individual clinicians. Frequently viewed and recurrently viewed topics might warrant special attention. Specialists usually view topics within their specialty and may have unique information needs.
Collapse
Affiliation(s)
- David A Cook
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, USA.,Mayo Clinic Online Learning, Mayo Clinic College of Medicine, Rochester, MN, USA.,Division of General Internal Medicine, Mayo Clinic
| | | | - Jane A Linderbaum
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, USA.,Department of Cardiovascular Diseases, Mayo Clinic
| | - Laurie J Pencille
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, USA.,Center for the Science of Health Care Delivery, Mayo Clinic
| | - Deborah J Rhodes
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, USA.,Division of Preventive Medicine, Mayo Clinic
| |
Collapse
|
9
|
Effectiveness of a clinical knowledge support system for reducing diagnostic errors in outpatient care in Japan: A retrospective study. Int J Med Inform 2017; 109:1-4. [PMID: 29195700 DOI: 10.1016/j.ijmedinf.2017.09.010] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2017] [Revised: 09/01/2017] [Accepted: 09/23/2017] [Indexed: 10/18/2022]
Abstract
Clinical evidence has indicated the effectiveness of computer-based systems for preventing and reducing diagnostic errors. Our study aimed to evaluate the effectiveness of UpToDate, a computer-based clinical knowledge management system, for reducing diagnostic errors. We retrospectively identified 100 patients who visited an outpatient department in a community-based hospital from July 2014 to June 2015. Fifty patients (exposure group) were seen by UpToDate-equipped physicians and another 50 (control group) were seen by UpToDate-unequipped physicians. We extracted data on patient sex, age, primary diagnosis, and case difficulty that could potentially affect diagnostic outcomes. We compared the two groups regarding diagnostic error rate and performed logistic regression analysis to analyze the concurrent effects of various factors affecting diagnostic error. The diagnostic error rate was 2% in the exposure group, while the error rate was 24% in the control group. Multivariate logistic regression analysis showed that error rate reduction was significantly associated with exposure to UpToDate with an odds ratio of 15.21 (95% CI 1.86-124.36). Our results demonstrated the effectiveness of UpToDate for the prevention and reduction of diagnostic error.
Collapse
|
10
|
Teixeira M, Cook DA, Heale BSE, Del Fiol G. Optimization of infobutton design and Implementation: A systematic review. J Biomed Inform 2017; 74:10-19. [PMID: 28838801 DOI: 10.1016/j.jbi.2017.08.010] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2017] [Revised: 08/11/2017] [Accepted: 08/19/2017] [Indexed: 01/22/2023]
Abstract
OBJECTIVE Infobuttons are clinical decision tools embedded in the electronic health record that attempt to link clinical data with context sensitive knowledge resources. We systematically reviewed technical approaches that contribute to improved infobutton design, implementation and functionality. METHODS We searched databases including MEDLINE, EMBASE, and the Cochrane Library database from inception to March 1, 2016 for studies describing the use of infobuttons. We selected full review comparative studies, usability studies, and qualitative studies examining infobutton design and implementation. We abstracted usability measures such as user satisfaction, impact, and efficiency, as well as prediction accuracy of infobutton content retrieval algorithms and infobutton adoption/interoperability. RESULTS We found 82 original research studies on infobuttons. Twelve studies met criteria for detailed abstraction. These studies investigated infobutton interoperability (1 study); tools to help tailor infobutton functionality (1 study); interventions to improve user experience (7 studies); and interventions to improve content retrieval by improving prediction of relevant knowledge resources and information needs (3 studies). In-depth interviews with implementers showed the Health Level Seven (HL7) Infobutton standard to be simple and easy to implement. A usability study demonstrated the feasibility of a tool to help medical librarians tailor infobutton functionality. User experience studies showed that access to resources with which users are familiar increased user satisfaction ratings; and that links to specific subsections of drug monographs increased information seeking efficiency. However, none of the user experience improvements led to increased usage uptake. Recommender systems based on machine learning algorithms outperformed hand-crafted rules in the prediction of relevant resources and clinicians' information needs in a laboratory setting, but no studies were found using these techniques in clinical settings. Improved content indexing in one study led to improved content retrieval across three health care organizations. CONCLUSION Best practice technical approaches to ensure optimal infobutton functionality, design and implementation remain understudied. The HL7 Infobutton standard has supported wide adoption of infobutton functionality among clinical information systems and knowledge resources. Limited evidence supports infobutton enhancements such as links to specific subtopics, configuration of optimal resources for specific tasks and users, and improved indexing and content coverage. Further research is needed to investigate user experience improvements to increase infobutton use and effectiveness.
Collapse
Affiliation(s)
- Miguel Teixeira
- Department of Internal Medicine, Mayo Clinic, Rochester, MN, United States; Department of Anesthesiology, Mayo Clinic, Rochester, MN, United States.
| | - David A Cook
- Department of Internal Medicine, Mayo Clinic, Rochester, MN, United States; Mayo Clinic Online Learning, Mayo Clinic College of Medicine and Science, Rochester, MN, United States; Knowledge Delivery Center, Mayo Clinic College of Medicine and Science, Rochester, MN, United States.
| | - Bret S E Heale
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States.
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States.
| |
Collapse
|
11
|
Valtis YK, Rosenberg J, Bhandari S, Wachter K, Teichman M, Beauvais S, Weintraub R. Evidence-based medicine for all: what we can learn from a programme providing free access to an online clinical resource to health workers in resource-limited settings. BMJ Glob Health 2017; 1:e000041. [PMID: 28588926 PMCID: PMC5321332 DOI: 10.1136/bmjgh-2016-000041] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2016] [Revised: 04/03/2016] [Accepted: 04/21/2016] [Indexed: 11/04/2022] Open
Abstract
The rapidly changing landscape of medical knowledge and guidelines requires health professionals to have immediate access to current, reliable clinical resources. Access to evidence is instrumental in reducing diagnostic errors and generating better health outcomes. UpToDate, a leading evidence-based clinical resource is used extensively in the USA and other regions of the world and has been linked to lower mortality and length of stay in US hospitals. In 2009, the Global Health Delivery Project collaborated with UpToDate to provide free subscriptions to qualifying health workers in resource-limited settings. We evaluated the provision of UpToDate access to health workers by analysing their usage patterns. Since 2009, ∼2000 individual physicians and healthcare institutions from 116 countries have received free access to UpToDate through our programme. During 2013-2014, users logged into UpToDate ∼150 000 times; 61% of users logged in at least weekly; users in Africa were responsible for 54% of the total usage. Search patterns reflected local epidemiology with 'clinical manifestations of malaria' as the top search in Africa, and 'management of hepatitis B' as the top search in Asia. Our programme demonstrates that there are barriers to evidence-based clinical knowledge in resource-limited settings we can help remove. Some assumed barriers to its expansion (poor internet connectivity, lack of training and infrastructure) might pose less of a burden than subscription fees.
Collapse
Affiliation(s)
- Yannis K Valtis
- Harvard Medical School, Boston, Massachusetts, USA.,Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Julie Rosenberg
- Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Sudip Bhandari
- Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Keri Wachter
- Harvard Medical School, Boston, Massachusetts, USA.,Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Marie Teichman
- Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Sophie Beauvais
- Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA
| | - Rebecca Weintraub
- Harvard Medical School, Boston, Massachusetts, USA.,Global Health Delivery Project at Harvard University, Cambridge, Massachusetts, USA.,Brigham and Women's Hospital, Boston, Massachusetts, USA
| |
Collapse
|
12
|
Cook DA, Teixeira MT, Heale BSE, Cimino JJ, Del Fiol G. Context-sensitive decision support (infobuttons) in electronic health records: a systematic review. J Am Med Inform Assoc 2017; 24:460-468. [PMID: 27497794 PMCID: PMC6080678 DOI: 10.1093/jamia/ocw104] [Citation(s) in RCA: 26] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2016] [Revised: 05/05/2016] [Accepted: 05/27/2016] [Indexed: 11/12/2022] Open
Abstract
OBJECTIVE Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes. METHODS We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information. RESULTS Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes. CONCLUSIONS Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.
Collapse
Affiliation(s)
- David A Cook
- Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, MN, USA
- Mayo Clinic Online Learning, Mayo Clinic College of Medicine, Rochester, MN, USA
- Department of Medicine, Mayo Clinic, Rochester, MN, USA
| | | | - Bret SE Heale
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| | - James J Cimino
- Informatics Institute, School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA
| | - Guilherme Del Fiol
- Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA
| |
Collapse
|
13
|
Hale LS, Wallace MM, Adams CR, Kaufman ML, Snyder CL. Considering Point-of-Care Electronic Medical Resources in Lieu of Traditional Textbooks for Medical Education. J Physician Assist Educ 2015; 26:161-166. [PMID: 26309211 DOI: 10.1097/jpa.0000000000000035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
Selecting resources to support didactic courses is a critical decision, and the advantages and disadvantages must be carefully considered. During clinical rotations, students not only need to possess strong background knowledge but also are expected to be proficient with the same evidence-based POC resources used by clinicians. Students place high value on “real world” learning and therefore may place more value on POC resources that they know practicing clinicians use as compared with medical textbooks. The condensed nature of PA education requires students to develop background knowledge and information literacy skills over a short period. One way to build that knowledge and those skills simultaneously is to use POC resources in lieu of traditional medical textbooks during didactic training. Electronic POC resources offer several advantages over traditional textbooks and should be considered as viable options in PA education.
Collapse
Affiliation(s)
- LaDonna S Hale
- LaDonna S. Hale, PharmD, is a professor in the Department of Physician Assistant Studies, Wichita State University, Wichita, Kansas. Michelle M. Wallace, MPAS, PA-C, is a clinical educator in the Department of Physician Assistant Studies, Wichita State University, Wichita, Kansas. Courtney R. Adams, BS, is a Master of Physician Assistant Studies student at Wichita State University, Wichita, Kansas. Michelle L. Kaufman, BS, is a Master of Physician Assistant Studies student at Wichita State University, Wichita, Kansas. Courtney L. Snyder, BS, is a Master of Physician Assistant Studies student at Wichita State University, Wichita, Kansas
| | | | | | | | | |
Collapse
|
14
|
Rakinic J, Buie WD. Maintenance of certification in 2015: A view from colon and rectal surgery. SEMINARS IN COLON AND RECTAL SURGERY 2015. [DOI: 10.1053/j.scrs.2015.04.010] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
15
|
Cook DA, Enders F, Caraballo PJ, Nishimura RA, Lloyd FJ. An automated clinical alert system for newly-diagnosed atrial fibrillation. PLoS One 2015; 10:e0122153. [PMID: 25849969 PMCID: PMC4388495 DOI: 10.1371/journal.pone.0122153] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2014] [Accepted: 02/13/2015] [Indexed: 11/18/2022] Open
Abstract
OBJECTIVE Clinical decision support systems that notify providers of abnormal test results have produced mixed results. We sought to develop, implement, and evaluate the impact of a computer-based clinical alert system intended to improve atrial fibrillation stroke prophylaxis, and identify reasons providers do not implement a guideline-concordant response. MATERIALS AND METHODS We conducted a cohort study with historical controls among patients at a tertiary care hospital. We developed a decision rule to identify newly-diagnosed atrial fibrillation, automatically notify providers, and direct them to online evidence-based management guidelines. We tracked all notifications from December 2009 to February 2010 (notification period) and applied the same decision rule to all patients from December 2008 to February 2009 (control period). Primary outcomes were accuracy of notification (confirmed through chart review) and prescription of warfarin within 30 days. RESULTS During the notification period 604 notifications were triggered, of which 268 (44%) were confirmed as newly-diagnosed atrial fibrillation. The notifications not confirmed as newly-diagnosed involved patients with no recent electrocardiogram at our institution. Thirty-four of 125 high-risk patients (27%) received warfarin in the notification period, compared with 34 of 94 (36%) in the control period (odds ratio, 0.66 [95% CI, 0.37-1.17]; p = 0.16). Common reasons to not prescribe warfarin (identified from chart review of 151 patients) included upcoming surgical procedure, choice to use aspirin, and discrepancy between clinical notes and the medication record. CONCLUSIONS An automated system to identify newly-diagnosed atrial fibrillation, notify providers, and encourage access to management guidelines did not change provider behaviors.
Collapse
Affiliation(s)
- David A. Cook
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, United States of America
- Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, United States of America
- Mayo Clinic Online Learning, Mayo Clinic College of Medicine, Rochester, MN, United States of America
| | - Felicity Enders
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, United States of America
- Department of Health Sciences Research, Mayo Clinic College of Medicine, Rochester, MN, United States of America
| | - Pedro J. Caraballo
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, United States of America
- Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, United States of America
| | - Rick A. Nishimura
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, United States of America
- Division of Cardiovascular Diseases, Mayo Clinic College of Medicine, Rochester, MN, United States of America
| | - Farrell J. Lloyd
- Knowledge Delivery Center, Mayo Clinic, Rochester, MN, United States of America
- Division of Hospital Medicine, Mayo Clinic College of Medicine, Rochester, MN, United States of America
| |
Collapse
|
16
|
Predictors of performance on the Maintenance of Certification in Anesthesiology Program® (MOCA®) examination. J Clin Anesth 2015; 27:1-6. [DOI: 10.1016/j.jclinane.2014.08.007] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2014] [Revised: 08/07/2014] [Accepted: 08/08/2014] [Indexed: 11/21/2022]
|
17
|
Cook DA, Sorensen KJ, Nishimura RA, Ommen SR, Lloyd FJ. A comprehensive information technology system to support physician learning at the point of care. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2015; 90:33-39. [PMID: 25374037 DOI: 10.1097/acm.0000000000000551] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
Abstract
MayoExpert is a multifaceted information system integrated with the electronic medical record (EMR) across Mayo Clinic's multisite health system. It was developed as a technology-based solution to manage information, standardize clinical practice, and promote and document learning in clinical contexts. Features include urgent test result notifications; models illustrating expert-approved care processes; concise, expert-approved answers to frequently asked questions (FAQs); a directory of topic-specific experts; and a portfolio for provider licensure and credentialing. The authors evaluate MayoExpert's reach, effectiveness, adoption, implementation, and maintenance. Evaluation data sources included usage statistics, user surveys, and pilot studies.As of October 2013, MayoExpert was available at 94 clinical sites in 12 states and contained 1,368 clinical topics, answers to 7,640 FAQs, and 92 care process models. In 2012, MayoExpert was accessed at least once by 2,578/3,643 (71%) staff physicians, 900/1,374 (66%) midlevel providers, and 1,728/2,291 (75%) residents and fellows. In a 2013 survey of MayoExpert users with 536 respondents, all features were highly rated (≥67% favorable). More providers reported using MayoExpert to answer questions before/after than during patient visits (68% versus 36%). During November 2012 to April 2013, MayoExpert sent 1,660 notifications of new-onset atrial fibrillation and 1,590 notifications of prolonged QT. MayoExpert has become part of routine clinical and educational operations, and its care process models now define Mayo Clinic best practices. MayoExpert's infrastructure and content will continue to expand with improved templates and content organization, new care process models, additional notifications, better EMR integration, and improved support for credentialing activities.
Collapse
Affiliation(s)
- David A Cook
- Dr. Cook is professor of medicine and medical education, Department of Medicine, associate director, Mayo Clinic Online Learning, and researcher, Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, Minnesota. Ms. Sorensen is head, Knowledge Management Technologies Unit, Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, Minnesota. Dr. Nishimura is professor of medicine, Department of Medicine, and Terrance D. and Judith A. Paul Director of MayoExpert, Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, Minnesota. Dr. Ommen is professor of medicine, Department of Medicine, and medical editor and MayoExpert physician lead, Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, Minnesota. Dr. Lloyd is assistant professor of medicine, Department of Medicine, and medical director, Wisdom Lab, Knowledge Delivery Center, Mayo Clinic College of Medicine, Rochester, Minnesota
| | | | | | | | | |
Collapse
|
18
|
Klingensmith ME, Jones AT, Smiley W, Biester TW, Malangoni MA. Subscription to the Surgical Council on Resident Education Web Portal and Qualifying Examination Performance. J Am Coll Surg 2014; 218:566-70. [DOI: 10.1016/j.jamcollsurg.2013.12.015] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2013] [Accepted: 12/10/2013] [Indexed: 11/25/2022]
|
19
|
Flannery MT. Recent trends in internal medicine education: a brief update. Eur J Intern Med 2014; 25:221-3. [PMID: 24559532 DOI: 10.1016/j.ejim.2014.01.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/18/2013] [Revised: 01/21/2014] [Accepted: 01/28/2014] [Indexed: 11/15/2022]
Abstract
This perspective attempts to bring graduate medical offices, residency programs and medical students interested in categorical internal medicine (CIM) a brief update on the American Board of Internal Medicine (ABIM), Accreditation Council for Graduate Medical Education (ACGME) and the National Registry for Medical Programs (NRMP) changes for the past 3-5 years in the United States (US). The US model for certification and recertification may serve as a homogenous example for other countries. This model will be described so that there is an understanding of the importance of such changes in the American system and its effect on resident education. This is critical knowledge for both teachers and learners in internal medicine in preparation for a lifetime career and requirements for certification/credentialing for both programs and their residents/fellows. Data from the review indicate a small increase in the number of applicants but a concordant decrease in ABIM initial certification exams. Programs should well be aware of the new focus on outcomes via the Next Accreditation System (NAS) being put forth by the ACGME.
Collapse
Affiliation(s)
- Michael T Flannery
- 12901 Bruce B Downs Blvd., MDC Box 19 Room L1041, Tampa, FL 33612, United States.
| |
Collapse
|
20
|
Abstract
AbstractThe last century saw dramatic changes in clinical practice and medical education and the concomitant rise in high-stakes, psychometrically-based examinations of medical knowledge. Higher scores on these high-stakes “in-vitro” examinations are modestly associated with better performance in clinical practice and provide a meaningful degree of assurance to the public about physicians’ competency in medical knowledge. However, results on such examinations explain only a small fraction of the wide variation currently seen in clinical practice and diagnostic errors remain a serious and vexing problem for patients and the healthcare system despite decades of high-stakes examinations. In this commentary we explore some of the limitations of high-stakes examinations in assessing clinical reasoning and propose utilizing situated cognition theory to guide research and development of innovative modes of ”in-vivo” assessments that can be used in longitudinally and continuously in clinical practice.
Collapse
|
21
|
Cook DA, Sorensen KJ, Hersh W, Berger RA, Wilkinson JM. Features of effective medical knowledge resources to support point of care learning: a focus group study. PLoS One 2013; 8:e80318. [PMID: 24282535 PMCID: PMC3840020 DOI: 10.1371/journal.pone.0080318] [Citation(s) in RCA: 39] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2013] [Accepted: 10/08/2013] [Indexed: 11/19/2022] Open
Abstract
OBJECTIVE Health care professionals access various information sources to quickly answer questions that arise in clinical practice. The features that favorably influence the selection and use of knowledge resources remain unclear. We sought to better understand how clinicians select among the various knowledge resources available to them, and from this to derive a model for an effective knowledge resource. METHODS We conducted 11 focus groups at an academic medical center and outlying community sites. We included a purposive sample of 50 primary care and subspecialist internal medicine and family medicine physicians. We transcribed focus group discussions and analyzed these using a constant comparative approach to inductively identify features that influence the selection of knowledge resources. RESULTS We identified nine features that influence users' selection of knowledge resources, namely efficiency (with sub-features of comprehensiveness, searchability, and brevity), integration with clinical workflow, credibility, user familiarity, capacity to identify a human expert, reflection of local care processes, optimization for the clinical question (e.g., diagnosis, treatment options, drug side effect), currency, and ability to support patient education. No single existing resource exemplifies all of these features. CONCLUSION The influential features identified in this study will inform the development of knowledge resources, and could serve as a framework for future research in this field.
Collapse
Affiliation(s)
- David A. Cook
- Office of Education Research, Mayo Medical School, Rochester, Minnesota, United States of America
- Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota, United States of America
- Knowledge Delivery Center, Mayo Clinic, Rochester, Minnesota, United States of America
| | - Kristi J. Sorensen
- Knowledge Delivery Center, Mayo Clinic, Rochester, Minnesota, United States of America
| | - William Hersh
- Department of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, United States of America
| | - Richard A. Berger
- School of Continuous Professional Development, Mayo Clinic College of Medicine, Rochester, Minnesota, United States of America
- Department of Orthopedic Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota, United States of America
| | - John M. Wilkinson
- Department of Family Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota, United States of America
| |
Collapse
|
22
|
|
23
|
Durning SJ, Ratcliffe T, Artino AR, van der Vleuten C, Beckman TJ, Holmboe E, Lipner RS, Schuwirth L. How is clinical reasoning developed, maintained, and objectively assessed? Views from expert internists and internal medicine interns. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2013; 33:215-223. [PMID: 24347100 DOI: 10.1002/chp.21188] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
INTRODUCTION There is limited understanding of how clinical reasoning is developed, maintained, and objectively assessed. Using the theoretical lens of deliberate practice, we explored physicians' views on clinical reasoning. We compared responses from internists (faculty) and internal medicine interns, to identify potential qualitative and/or quantitative differences in how clinical reasoning is developed and maintained. METHODS Participants' free-text comments regarding how clinical reasoning is developed, maintained, and objectively assessed were analyzed. Three investigators coded responses using a constant-comparative, grounded theory approach. We also compared the frequencies of each theme between the 2 groups. RESULTS Twenty-two faculty and 17 interns participated in this study. Faculty and intern themes for how clinical reasoning is developed, maintained, and objectively assessed were similar, but quantitative and qualitative differences emerged. Interrater reliability of themes was high (overall kappa: 0.92; range: 0.88-0.98). Only experts (faculty) mentioned the value of teaching for development and maintenance of clinical reasoning. Interns focused on knowledge acquisition activities and use of online resources. Experts and intern participants both struggled with how to best measure clinical reasoning; direct observation was rarely mentioned as a strategy. DISCUSSION Consistent with our theoretical expectations, we found quantitative and qualitative differences in participants' responses, which have implications for teaching and assessment of clinical reasoning. By capturing the types of activities and their relative frequencies within and between these groups, this work adds to the deliberate practice literature.
Collapse
Affiliation(s)
- Steven J Durning
- Professor of Medicine and Pathology, Uniformed Services University
| | | | | | | | | | | | | | | |
Collapse
|
24
|
Lipner RS, Hess BJ, Phillips RL. Specialty board certification in the United States: issues and evidence. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2013; 33 Suppl 1:S20-S35. [PMID: 24347150 DOI: 10.1002/chp.21203] [Citation(s) in RCA: 77] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/03/2023]
Abstract
BACKGROUND The American Board of Medical Specialties (ABMS) certification and maintenance of certification (MOC) programs strive to provide the public with guidance about a physician's competence. This study summarizes the literature on the effectiveness of these programs. METHOD A literature search was conducted for studies published between 1986 and April 2013 and limited to ABMS certification. A modified version of Kirkpatrick's 4 levels of program evaluation included the reaction of stakeholders to certification, the extent to which physicians are encouraged to improve, the relationship between performance in the programs and nonclinical external measures of physician competence, and the relationship of performance in the programs with clinical quality measures. RESULTS Patients' and hospitals' value of board certification and physician participation in MOC are high. Physicians are conflicted as to whether the effort involved is worth its value. Self-reported evidence shows improvement in knowledge, practice infrastructure, communication with patients and peers, and clinical care. Certification performance is generally related to nonclinical external measures such as types of training, practice characteristics, demographics, and disciplinary actions. In general, physicians who are board certified provide better patient care, albeit the results have modest effect sizes and are not unequivocal. CONCLUSIONS Certification boards should continuously try to improve their programs in response to feedback from stakeholders, changes in the way physicians practice, as well as the growth in the fields of measurement and technology. Keeping pace with these changes in a responsible and evidence-based way is important.
Collapse
Affiliation(s)
- Rebecca S Lipner
- Senior Vice President, Evaluation, Research & Development, American Board of Internal Medicine.
| | | | | |
Collapse
|
25
|
Norcini JJ, Lipner RS, Grosso LJ. Assessment in the context of licensure and certification. TEACHING AND LEARNING IN MEDICINE 2013; 25 Suppl 1:S62-S67. [PMID: 24246109 DOI: 10.1080/10401334.2013.842909] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
Over the past 25 years, three major forces have had a significant influence on licensure and certification: the shift in focus from educational process to educational outcomes, the increasing recognition of the need for learning and assessment throughout a physician's career, and the changes in technology and psychometrics that have opened new vistas for assessment. These forces have led to significant changes in assessment for licensure and certification. To respond to these forces, licensure and certification programs have improved the ways in which their examinations are constructed, scored, and delivered. In particular, we note the introduction of adaptive testing; automated item creation, scoring, and test assembly; assessment engineering; and data forensics. Licensure and certification programs have also expanded their repertoire of assessments with the rapid development and adoption of simulation and workplace-based assessment. Finally, they have invested in research intended to validate their programs in four ways: (a) the acceptability of the program to stakeholders, (b) the extent to which stakeholders are encouraged to learn and improve, (c) the extent to which there is a relationship between performance in the programs and external measures, and (d) the extent to which there is a relationship between performance as measured by the assessment and performance in practice. Over the past 25 years, changes in licensure and certification have been driven by the educational outcomes movement, the need for lifelong learning, and advances in technology and psychometrics. Over the next 25 years, we expect these forces to continue to exert pressure for change which will lead to additional improvement and expansion in examination processes, methods of assessment, and validation research.
Collapse
Affiliation(s)
- John J Norcini
- a Foundation for Advancement of International Medical Education and Research , Philadelphia , Pennsylvania , USA
| | | | | |
Collapse
|