Effectiveness of topic-specific infobuttons: a randomized controlled trial

Ontologies Applied in Clinical Decision Support System Rules: Systematic Review

BACKGROUND

Clinical decision support systems (CDSSs) are important for the quality and safety of health care delivery. Although CDSS rules guide CDSS behavior, they are not routinely shared and reused.

OBJECTIVE

Ontologies have the potential to promote the reuse of CDSS rules. Therefore, we systematically screened the literature to elaborate on the current status of ontologies applied in CDSS rules, such as rule management, which uses captured CDSS rule usage data and user feedback data to tailor CDSS services to be more accurate, and maintenance, which updates CDSS rules. Through this systematic literature review, we aim to identify the frontiers of ontologies used in CDSS rules.

METHODS

The literature search was focused on the intersection of ontologies; clinical decision support; and rules in PubMed, the Association for Computing Machinery (ACM) Digital Library, and the Nursing & Allied Health Database. Grounded theory and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines were followed. One author initiated the screening and literature review, while 2 authors validated the processes and results independently. The inclusion and exclusion criteria were developed and refined iteratively.

RESULTS

CDSSs were primarily used to manage chronic conditions, alerts for medication prescriptions, reminders for immunizations and preventive services, diagnoses, and treatment recommendations among 81 included publications. The CDSS rules were presented in Semantic Web Rule Language, Jess, or Jena formats. Despite the fact that ontologies have been used to provide medical knowledge, CDSS rules, and terminologies, they have not been used in CDSS rule management or to facilitate the reuse of CDSS rules.

CONCLUSIONS

Ontologies have been used to organize and represent medical knowledge, controlled vocabularies, and the content of CDSS rules. So far, there has been little reuse of CDSS rules. More work is needed to improve the reusability and interoperability of CDSS rules. This review identified and described the ontologies that, despite their limitations, enable Semantic Web technologies and their applications in CDSS rules.

Feeling and thinking: can theories of human motivation explain how EHR design impacts clinician burnout?

The psychology of motivation can help us understand the impact of electronic health records (EHRs) on clinician burnout both directly and indirectly. Informatics approaches to EHR usability tend to focus on the extrinsic motivation associated with successful completion of clearly defined tasks in clinical workflows. Intrinsic motivation, which includes the need for autonomy, sense-making, creativity, connectedness, and mastery is not well supported by current designs and workflows. This piece examines existing research on the importance of 3 psychological drives in relation to healthcare technology: goal-based decision-making, sense-making, and agency/autonomy. Because these motives are ubiquitous, foundational to human functioning, automatic, and unconscious, they may be overlooked in technological interventions. The results are increased cognitive load, emotional distress, and unfulfilling workplace environments. Ultimately, we hope to stimulate new research on EHR design focused on expanding functionality to support intrinsic motivation, which, in turn, would decrease burnout and improve care.

Regenstrief teaching electronic medical record (tEMR) platform: a novel tool for teaching and evaluating applied health information technology

The objective of this study is to provide an overview of the Regenstrief Teaching Electronic Medical Record (tEMR), how the tEMR could be used, and how it is currently being used in health professions education. The tEMR is a derivative of a real-world electronic health record (EHR), a large, pseudonymized patient database, and a population health tool designed to support curricular goals. The tEMR has been successfully adopted at 12 health professional, public health, and health information technology (HIT) schools, with over 11 800 unique student users and more than 74 000 logins, for case presentation, to develop diagnostic and therapeutic plans, and to practice documentation skills. With the exponential growth of health-related data and the impact of HIT on work-life balance, it is critical for students to get early EHR skills practice and understand how EHR's work. The tEMR is a promising, scalable, flexible application to help health professional students learn about common HIT tools and issues.

Evaluating the Impact of the Grading and Assessment of Predictive Tools Framework on Clinicians and Health Care Professionals' Decisions in Selecting Clinical Predictive Tools: Randomized Controlled Trial

BACKGROUND

While selecting predictive tools for implementation in clinical practice or for recommendation in clinical guidelines, clinicians and health care professionals are challenged with an overwhelming number of tools. Many of these tools have never been implemented or evaluated for comparative effectiveness. To overcome this challenge, the authors developed and validated an evidence-based framework for grading and assessment of predictive tools (the GRASP framework). This framework was based on the critical appraisal of the published evidence on such tools.

OBJECTIVE

The aim of the study was to examine the impact of using the GRASP framework on clinicians' and health care professionals' decisions in selecting clinical predictive tools.

METHODS

A controlled experiment was conducted through a web-based survey. Participants were randomized to either review the derivation publications, such as studies describing the development of the predictive tools, on common traumatic brain injury predictive tools (control group) or to review an evidence-based summary, where each tool had been graded and assessed using the GRASP framework (intervention group). Participants in both groups were asked to select the best tool based on the greatest validation or implementation. A wide group of international clinicians and health care professionals were invited to participate in the survey. Task completion time, rate of correct decisions, rate of objective versus subjective decisions, and level of decisional conflict were measured.

RESULTS

We received a total of 194 valid responses. In comparison with not using GRASP, using the framework significantly increased correct decisions by 64%, from 53.7% to 88.1% (88.1/53.7=1.64; t₁₉₃=8.53; P<.001); increased objective decision making by 32%, from 62% (3.11/5) to 82% (4.10/5; t₁₈₉=9.24; P<.001); decreased subjective decision making based on guessing by 20%, from 49% (2.48/5) to 39% (1.98/5; t₁₈₈=-5.47; P<.001); and decreased prior knowledge or experience by 8%, from 71% (3.55/5) to 65% (3.27/5; t₁₈₇=-2.99; P=.003). Using GRASP significantly decreased decisional conflict and increased the confidence and satisfaction of participants with their decisions by 11%, from 71% (3.55/5) to 79% (3.96/5; t₁₈₈=4.27; P<.001), and by 13%, from 70% (3.54/5) to 79% (3.99/5; t₁₈₈=4.89; P<.001), respectively. Using GRASP decreased the task completion time, on the 90th percentile, by 52%, from 12.4 to 6.4 min (t₁₉₃=-0.87; P=.38). The average System Usability Scale of the GRASP framework was very good: 72.5% and 88% (108/122) of the participants found the GRASP useful.

CONCLUSIONS

Using GRASP has positively supported and significantly improved evidence-based decision making. It has increased the accuracy and efficiency of selecting predictive tools. GRASP is not meant to be prescriptive; it represents a high-level approach and an effective, evidence-based, and comprehensive yet simple and feasible method to evaluate, compare, and select clinical predictive tools.

Effect of a clinical evidence technology on patient skin disease outcomes in primary care: a cluster-randomized controlled trial

Objective

Providers' use of clinical evidence technologies (CETs) improves their diagnosis and treatment decisions. Despite these benefits, few studies have evaluated the impact of CETs on patient outcomes. The investigators evaluated the effect of one CET, VisualDx, on skin problem outcomes in primary care.

Methods

A cluster-randomized controlled pragmatic trial was conducted in outpatient clinics at an academic medical center in the northeastern United States. Participants were primary care providers (PCPs) and their adult patients seen for skin problems. The intervention was VisualDx, as used by PCPs. Outcomes were patient-reported time from index clinic visit to problem resolution, and the number of follow-up visits to any provider for the same problem. PCPs who were randomly assigned to the intervention agreed to use VisualDx as their primary evidence source for skin problems. Control group PCPs agreed not to use VisualDx. Investigators collected outcome data from patients by phone at thirty-day intervals. Cox proportional hazards models assessed time to resolution. Wilcoxon-rank sum tests and logistic regression compared the need for return appointments.

Results

Thirty-two PCPs and 433 patients participated. In proportional hazards modelling adjusted for provider clusters, the time from index visit to skin problem resolution was similar in both groups (hazard ratio=0.92; 95% confidence interval [CI]=0.70, 1.21; p=0.54). Patient follow-up appointments did not differ significantly between groups (odds ratio=1.26; CI=0.94, 1.70; p=0.29).

Conclusion

This pragmatic trial tested the effectiveness of VisualDx on patient-reported skin disease outcomes in a generalizable clinical setting. There was no difference in skin problem resolution or number of follow-up visits when PCPs used VisualDx.

Health Care Providers' Profiles and Evaluations of a Statewide Online Education Program for Dissemination of Clinical Evidence on HIV, Hepatitis C Virus, and Sexually Transmitted Disease: Cross-Sectional Study

BACKGROUND

Timely and effective dissemination of the latest clinical evidence to health care providers is essential for translating biomedical research into routine patient care. Online platforms offer unique opportunities for dissemination of medical knowledge.

OBJECTIVE

In this study, we report the profiles of health care providers participating in the New York State HIV-HCV-STD Clinical Education Initiative online program and their evaluations of the online continuing professional development courses.

METHODS

We compiled professional and personal background information of the clinicians who completed at least one online course. We collected their self-reported program evaluation data with regard to the course content, format, knowledge increase, and impact on clinical practice.

RESULTS

We recorded a total of 4363 completions of 88 online courses by 1976 unique clinicians during a 12-month study period. The clinicians' background was diverse in terms of demographics, education levels, professional disciplines, practice years, employment settings, caseloads, and clinical services. The evaluation of online courses was very positive (usefulness/relevance, 91.08%; easy comprehension, 89.09%; knowledgeable trainer, 92.00%; appropriate format, 84.35%; knowledge increase, 48.52%; intention to use knowledge, 85.26%; and plan to change practice, 21.98%). Comparison with the reference data indicated that the online program successfully reached out to the primary care communities. Both the younger generation and the senior health care providers were attracted to the online program. High-quality multimedia resources, flexibility of access, ease of use, and provision of continuing professional development credits contributed to the initial success of this online clinical education program.

CONCLUSIONS

We have successfully characterized a diverse group of clinicians participating in a statewide online continuing professional development program. The evaluation has shown effective use of online resources to disseminate clinical evidence on HIV, hepatitis C virus, and sexually transmitted disease to primary care clinicians.

A systematic review of trials evaluating success factors of interventions with computerised clinical decision support

BACKGROUND

Computerised clinical decision support (CDS) can potentially better inform decisions, and it can help with the management of information overload. It is perceived to be a key component of a learning health care system. Despite its increasing implementation worldwide, it remains uncertain why the effect of CDS varies and which factors make CDS more effective.

OBJECTIVE

To examine which factors make CDS strategies more effective on a number of outcomes, including adherence to recommended practice, patient outcome measures, economic measures, provider or patient satisfaction, and medical decision quality.

METHODS

We identified randomised controlled trials, non-randomised trials, and controlled before-and-after studies that directly compared CDS implementation with a given factor to CDS without that factor by searching CENTRAL, MEDLINE, EMBASE, and CINAHL and checking reference lists of relevant studies. We considered CDS with any objective for any condition in any healthcare setting. We included CDS interventions that were either displayed on screen or provided on paper and that were directed at healthcare professionals or targeted at both professionals and patients. The reviewers screened the potentially relevant studies in duplicate. They extracted data and assessed risk of bias in independent pairs or individually followed by a double check by another reviewer. We summarised results using medians and interquartile ranges and rated our certainty in the evidence using the GRADE system.

RESULTS

We identified 66 head-to-head trials that we synthesised across 14 comparisons of CDS intervention factors. Providing CDS automatically versus on demand led to large improvements in adherence. Displaying CDS on-screen versus on paper led to moderate improvements and making CDS more versus less patient-specific improved adherence modestly. When CDS interventions were combined with professional-oriented strategies, combined with patient-oriented strategies, or combined with staff-oriented strategies, then adherence improved slightly. Providing CDS to patients slightly increased adherence versus CDS aimed at the healthcare provider only. Making CDS advice more explicit and requiring users to respond to the advice made little or no difference. The CDS intervention factors made little or no difference to patient outcomes. The results for economic outcomes and satisfaction outcomes were sparse.

CONCLUSION

Multiple factors may affect the success of CDS interventions. CDS may be more effective when the advice is provided automatically and displayed on-screen and when the suggestions are more patient-specific. CDS interventions combined with other strategies probably also improves adherence. Providing CDS directly to patients may also positively affect adherence. The certainty of the evidence was low to moderate for all factors.

TRIAL REGISTRATION

PROSPERO, CRD42016033738.

Professional Development Perceptions and Practices Among U.S. Physicians: A Cross-Specialty National Survey

PURPOSE

Professional development (PD)-both for-credit continuing medical education (CME) and informal self-directed or point-of-care learning-is vital to all physicians. The authors sought to understand physicians' PD perceptions and practices and how these vary by specialty and practice type.

METHOD

The authors administered an Internet and paper survey, from September 2015 to April 2016, to randomly sampled U.S. physicians. Survey items addressed perceived PD needs and barriers and how physicians identify knowledge/skills gaps.

RESULTS

Of 4,648 invitees, 988 (21.6%) responded. Respondents believed that they already know what they need to learn (mean 5.8 [1 = strongly disagree; 7 = strongly agree]), can answer clinical questions using available resources (5.9), and want credit for learning during patient care (5.1). They did not strongly desire help identifying learning gaps (4.0) or indicate difficulty accumulating CME credits (3.1). Most PD was done during personal time (5.5). Competencies regarding medical knowledge/skills, wellness, informatics, and practice/systems improvement were rated the highest priority, while research, teaching, and professionalism were rated the lowest. The most important sources used to identify knowledge/skills gaps were immediate patient care needs (4.1 [1 = not important; 5 = extremely important]), personal awareness (3.8), and practice updates (3.7). The most important barriers were time (3.5) and cost (2.9). Differences by specialty and practice type were generally small and not statistically significant.

CONCLUSIONS

Physicians feel confident in identifying their own learning needs, perceive medical knowledge/skills as their highest-priority need, and desire more credit for learning during patient care.

Optimization of infobutton design and Implementation: A systematic review

OBJECTIVE

Infobuttons are clinical decision tools embedded in the electronic health record that attempt to link clinical data with context sensitive knowledge resources. We systematically reviewed technical approaches that contribute to improved infobutton design, implementation and functionality.

METHODS

We searched databases including MEDLINE, EMBASE, and the Cochrane Library database from inception to March 1, 2016 for studies describing the use of infobuttons. We selected full review comparative studies, usability studies, and qualitative studies examining infobutton design and implementation. We abstracted usability measures such as user satisfaction, impact, and efficiency, as well as prediction accuracy of infobutton content retrieval algorithms and infobutton adoption/interoperability.

RESULTS

We found 82 original research studies on infobuttons. Twelve studies met criteria for detailed abstraction. These studies investigated infobutton interoperability (1 study); tools to help tailor infobutton functionality (1 study); interventions to improve user experience (7 studies); and interventions to improve content retrieval by improving prediction of relevant knowledge resources and information needs (3 studies). In-depth interviews with implementers showed the Health Level Seven (HL7) Infobutton standard to be simple and easy to implement. A usability study demonstrated the feasibility of a tool to help medical librarians tailor infobutton functionality. User experience studies showed that access to resources with which users are familiar increased user satisfaction ratings; and that links to specific subsections of drug monographs increased information seeking efficiency. However, none of the user experience improvements led to increased usage uptake. Recommender systems based on machine learning algorithms outperformed hand-crafted rules in the prediction of relevant resources and clinicians' information needs in a laboratory setting, but no studies were found using these techniques in clinical settings. Improved content indexing in one study led to improved content retrieval across three health care organizations.

CONCLUSION

Best practice technical approaches to ensure optimal infobutton functionality, design and implementation remain understudied. The HL7 Infobutton standard has supported wide adoption of infobutton functionality among clinical information systems and knowledge resources. Limited evidence supports infobutton enhancements such as links to specific subtopics, configuration of optimal resources for specific tasks and users, and improved indexing and content coverage. Further research is needed to investigate user experience improvements to increase infobutton use and effectiveness.

Physicians' pharmacogenomics information needs and seeking behavior: a study with case vignettes

Background

Genetic testing, especially in pharmacogenomics, can have a major impact on patient care. However, most physicians do not feel that they have sufficient knowledge to apply pharmacogenomics to patient care. Online information resources can help address this gap. We investigated physicians’ pharmacogenomics information needs and information-seeking behavior, in order to guide the design of pharmacogenomics information resources that effectively meet clinical information needs.

Methods

We performed a formative, mixed-method assessment of physicians’ information-seeking process in three pharmacogenomics case vignettes. Interactions of 6 physicians’ with online pharmacogenomics resources were recorded, transcribed, and analyzed for prominent themes. Quantitative data included information-seeking duration, page navigations, and number of searches entered.

Results

We found that participants searched an average of 8 min per case vignette, spent less than 30 s reviewing specific content, and rarely refined search terms. Participants’ information needs included a need for clinically meaningful descriptions of test interpretations, a molecular basis for the clinical effect of drug variation, information on the logistics of carrying out a genetic test (including questions related to cost, availability, test turn-around time, insurance coverage, and accessibility of expert support).Also, participants sought alternative therapies that would not require genetic testing.

Conclusion

This study of pharmacogenomics information-seeking behavior indicates that content to support their information needs is dispersed and hard to find. Our results reveal a set of themes that information resources can use to help physicians find and apply pharmacogenomics information to the care of their patients.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0510-9) contains supplementary material, which is available to authorized users.

Context-sensitive decision support (infobuttons) in electronic health records: a systematic review

OBJECTIVE

Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes.

METHODS

We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information.

RESULTS

Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes.

CONCLUSIONS

Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.

Expanding access to high-quality plain-language patient education information through context-specific hyperlinks

Medical records, which are increasingly directly accessible to patients, contain highly technical terms unfamiliar to many patients. A federally qualified health center (FQHC) sought to help patients interpret their records by embedding context-specific hyperlinks to plain-language patient education materials in its portal. We assessed the impact of this innovation through a 3-year retrospective cohort study. A total of 12,877 (10% of all patients) in this safety net population had used the MPC links. Black patients, Latino patients comfortable using English, and patients covered by Medicaid were more likely to use the informational hyperlinks than other patients. The positive association with black race and Latino ethnicity remained statistically significant in multivariable models that controlled for insurance type. We conclude that many of the sociodemographic factors associated with the digital divide do not present barriers to accessing context-specific patient education information once in the portal. In fact, this type of highly convenient plain-language patient education may provide particular value to patients in traditionally disadvantaged groups.

Identifying complexity in infectious diseases inpatient settings: An observation study

BACKGROUND

Understanding complexity in healthcare has the potential to reduce decision and treatment uncertainty. Therefore, identifying both patient and task complexity may offer better task allocation and design recommendation for next-generation health information technology system design.

OBJECTIVE

To identify specific complexity-contributing factors in the infectious disease domain and the relationship with the complexity perceived by clinicians.

METHOD

We observed and audio recorded clinical rounds of three infectious disease teams. Thirty cases were observed for a period of four consecutive days. Transcripts were coded based on clinical complexity-contributing factors from the clinical complexity model. Ratings of complexity on day 1 for each case were collected. We then used statistical methods to identify complexity-contributing factors in relationship to perceived complexity of clinicians.

RESULTS

A factor analysis (principal component extraction with varimax rotation) of specific items revealed three factors (eigenvalues>2.0) explaining 47% of total variance, namely task interaction and goals (10 items, 26%, Cronbach's Alpha=0.87), urgency and acuity (6 items, 11%, Cronbach's Alpha=0.67), and psychosocial behavior (4 items, 10%, Cronbach's alpha=0.55). A linear regression analysis showed no statistically significant association between complexity perceived by the physicians and objective complexity, which was measured from coded transcripts by three clinicians (Multiple R-squared=0.13, p=0.61). There were no physician effects on the rating of perceived complexity.

CONCLUSION

Task complexity contributes significantly to overall complexity in the infectious diseases domain. The different complexity-contributing factors found in this study can guide health information technology system designers and researchers for intuitive design. Thus, decision support tools can help reduce the specific complexity-contributing factors. Future studies aimed at understanding clinical domain-specific complexity-contributing factors can ultimately improve task allocation and design for intuitive clinical reasoning.

Integrating Genomic Resources with Electronic Health Records using the HL7 Infobutton Standard

BACKGROUND

The Clinical Genome Resource (ClinGen) Electronic Health Record (EHR) Workgroup aims to integrate ClinGen resources with EHRs. A promising option to enable this integration is through the Health Level Seven (HL7) Infobutton Standard. EHR systems that are certified according to the US Meaningful Use program provide HL7-compliant infobutton capabilities, which can be leveraged to support clinical decision-making in genomics.

OBJECTIVES

To integrate genomic knowledge resources using the HL7 infobutton standard. Two tactics to achieve this objective were: (1) creating an HL7-compliant search interface for ClinGen, and (2) proposing guidance for genomic resources on achieving HL7 Infobutton standard accessibility and compliance.

METHODS

We built a search interface utilizing OpenInfobutton, an open source reference implementation of the HL7 Infobutton standard. ClinGen resources were assessed for readiness towards HL7 compliance. Finally, based upon our experiences we provide recommendations for publishers seeking to achieve HL7 compliance.

RESULTS

Eight genomic resources and two sub-resources were integrated with the ClinGen search engine via OpenInfobutton and the HL7 infobutton standard. Resources we assessed have varying levels of readiness towards HL7-compliance. Furthermore, we found that adoption of standard terminologies used by EHR systems is the main gap to achieve compliance.

CONCLUSION

Genomic resources can be integrated with EHR systems via the HL7 Infobutton standard using OpenInfobutton. Full compliance of genomic resources with the Infobutton standard would further enhance interoperability with EHR systems.

Rationale and study protocol for a multi-component Health Information Technology (HIT) screening tool for depression and post-traumatic stress disorder in the primary care setting

The prevalence rate of depression in primary care is high. Primary care providers serve as the initial point of contact for the majority of patients with depression, yet, approximately 50% of cases remain unrecognized. The under-diagnosis of depression may be further exacerbated in limited English-language proficient (LEP) populations. Language barriers may result in less discussion of patients' mental health needs and fewer referrals to mental health services, particularly given competing priorities of other medical conditions and providers' time pressures. Recent advances in Health Information Technology (HIT) may facilitate novel ways to screen for depression and other mental health disorders in LEP populations. The purpose of this paper is to describe the rationale and protocol of a clustered randomized controlled trial that will test the effectiveness of an HIT intervention that provides a multi-component approach to delivering culturally competent, mental health care in the primary care setting. The HIT intervention has four components: 1) web-based provider training, 2) multimedia electronic screening of depression and PTSD in the patients' primary language, 3) Computer generated risk assessment scores delivered directly to the provider, and 4) clinical decision support. The outcomes of the study include assessing the potential of the HIT intervention to improve screening rates, clinical detection, provider initiation of treatment, and patient outcomes for depression and post-traumatic stress disorder (PTSD) among LEP Cambodian refugees who experienced war atrocities and trauma during the Khmer Rouge. This technology has the potential to be adapted to any LEP population in order to facilitate mental health screening and treatment in the primary care setting.

Classification of clinically useful sentences in clinical evidence resources

Most patient care questions raised by clinicians can be answered by online clinical knowledge resources. However, important barriers still challenge the use of these resources at the point of care.

OBJECTIVE

To design and assess a method for extracting clinically useful sentences from synthesized online clinical resources that represent the most clinically useful information for directly answering clinicians' information needs.

MATERIALS AND METHODS

We developed a Kernel-based Bayesian Network classification model based on different domain-specific feature types extracted from sentences in a gold standard composed of 18 UpToDate documents. These features included UMLS concepts and their semantic groups, semantic predications extracted by SemRep, patient population identified by a pattern-based natural language processing (NLP) algorithm, and cue words extracted by a feature selection technique. Algorithm performance was measured in terms of precision, recall, and F-measure.

RESULTS

The feature-rich approach yielded an F-measure of 74% versus 37% for a feature co-occurrence method (p<0.001). Excluding predication, population, semantic concept or text-based features reduced the F-measure to 62%, 66%, 58% and 69% respectively (p<0.01). The classifier applied to Medline sentences reached an F-measure of 73%, which is equivalent to the performance of the classifier on UpToDate sentences (p=0.62).

CONCLUSIONS

The feature-rich approach significantly outperformed general baseline methods. This approach significantly outperformed classifiers based on a single type of feature. Different types of semantic features provided a unique contribution to overall classification performance. The classifier's model and features used for UpToDate generalized well to Medline abstracts.

Clinical management IT system for enhanced recovery

Surgical resection has a key role for the treatment of early stage lung cancer along with certain advanced cases, and minimally invasive techniques, representatively video-assisted thoracoscopic surgery (VATS), are becoming standard for lung cancer surgery. Implementation of integrated programs which could manage the whole process of patient treatment including preoperative, intraoperative and postoperative care is thought to be essential partner for successful application of minimally invasive thoracic surgery for lung cancer treatment. Enhanced recovery after surgery (ERAS), so called "fast-track" programs pursue the adequate and efficient delivery of health care services therefore to improve postoperative outcomes and reduce medical cost. Well-organized information technology systems would be helpful to achieve the goals of ERAS without increasing the burden of budget or working staffs. Furthermore, it could contribute to create knowledge and translate to the clinical process.

Usability and Acceptance of the Librarian Infobutton Tailoring Environment: An Open Access Online Knowledge Capture, Management, and Configuration Tool for OpenInfobutton

BACKGROUND

The Librarian Infobutton Tailoring Environment (LITE) is a Web-based knowledge capture, management, and configuration tool with which users can build profiles used by OpenInfobutton, an open source infobutton manager, to provide electronic health record users with context-relevant links to online knowledge resources.

OBJECTIVE

We conducted a multipart evaluation study to explore users' attitudes and acceptance of LITE and to guide future development.

METHODS

The evaluation consisted of an initial online survey to all LITE users, followed by an observational study of a subset of users in which evaluators' sessions were recorded while they conducted assigned tasks. The observational study was followed by administration of a modified System Usability Scale (SUS) survey.

RESULTS

Fourteen users responded to the survey and indicated good acceptance of LITE with feedback that was mostly positive. Six users participated in the observational study, demonstrating average task completion time of less than 6 minutes and an average SUS score of 72, which is considered good compared with other SUS scores.

CONCLUSIONS

LITE can be used to fulfill its designated tasks quickly and successfully. Evaluators proposed suggestions for improvements in LITE functionality and user interface.

Formative evaluation of a patient-specific clinical knowledge summarization tool

OBJECTIVE

To iteratively design a prototype of a computerized clinical knowledge summarization (CKS) tool aimed at helping clinicians finding answers to their clinical questions; and to conduct a formative assessment of the usability, usefulness, efficiency, and impact of the CKS prototype on physicians' perceived decision quality compared with standard search of UpToDate and PubMed.

MATERIALS AND METHODS

Mixed-methods observations of the interactions of 10 physicians with the CKS prototype vs. standard search in an effort to solve clinical problems posed as case vignettes.

RESULTS

The CKS tool automatically summarizes patient-specific and actionable clinical recommendations from PubMed (high quality randomized controlled trials and systematic reviews) and UpToDate. Two thirds of the study participants completed 15 out of 17 usability tasks. The median time to task completion was less than 10s for 12 of the 17 tasks. The difference in search time between the CKS and standard search was not significant (median=4.9 vs. 4.5m in). Physician's perceived decision quality was significantly higher with the CKS than with manual search (mean=16.6 vs. 14.4; p=0.036).

CONCLUSIONS

The CKS prototype was well-accepted by physicians both in terms of usability and usefulness. Physicians perceived better decision quality with the CKS prototype compared to standard search of PubMed and UpToDate within a similar search time. Due to the formative nature of this study and a small sample size, conclusions regarding efficiency and efficacy are exploratory.

Analysis of empty responses from electronic resources in infobutton managers

Infobuttons provide context-aware educational materials to both providers and patients and are becoming an important element in modern electronic health records (EHR) and patient health records (PHR). However, the content from different electronic resources (e-resource) as responses from infobutton manager has not been fully analyzed and evaluated. In this paper, we propose a method for automatically analyzing responses from infobutton manager. A tool is implemented to retrieve and analyze responses from infobutton manager. To test the tool, we extracted and sampled common and uncommon concepts from EHR usage data in Intermountain Healthcare's enterprise data warehouse. From the output of the tool, we evaluate infobutton performance by multiple categories, including against the most and less common used concepts, grouped by different modules in patient portal, by different e-resources, and by type of access (standardized Health Level Seven (HL7) vs not). Based on the results of our evaluation, we provide suggestions for further enhancements of infobuttons to the current implementation, including suggesting accessing priorities of e-resources and encouraging the use of the HL7 standard.

eHealth 2015 Special Issue: Impact of Electronic Health Records on the Completeness of Clinical Documentation Generated during Diabetic Retinopathy Consultations

BACKGROUND

Two years ago, the Diabetic Retinopathy (DRP) and Traumatology clinic of the Department of Ophthalmology and Optometrics at the Medical University of Vienna, Austria switched from paper-based to electronic health records. A customized electronic health record system (EHR-S) was implemented.

OBJECTIVES

To assess the completeness of information documented electronically compared with manually during patient visits.

METHODS

The Preferred Practice Pattern for Diabetic Retinopathy published by the American Academy of Ophthalmology was distilled into a list of medical features grouped into categories to be assessed and documented during the management of patients with DRP. The last seventy paper-based records and all electronic records generated since the switch were analyzed and graded for the presence of features on the list and the resulting scores compared.

RESULTS

In all categories, clinical documentation was more complete in the EHR group.

CONCLUSIONS

In our setting, the implementation of an EHR-S showed a statistically significant positive impact on documentation completeness.

Using natural language processing to provide personalized learning opportunities from trainee clinical notes

OBJECTIVE

Assessment of medical trainee learning through pre-defined competencies is now commonplace in schools of medicine. We describe a novel electronic advisor system using natural language processing (NLP) to identify two geriatric medicine competencies from medical student clinical notes in the electronic medical record: advance directives (AD) and altered mental status (AMS).

MATERIALS AND METHODS

Clinical notes from third year medical students were processed using a general-purpose NLP system to identify biomedical concepts and their section context. The system analyzed these notes for relevance to AD or AMS and generated custom email alerts to students with embedded supplemental learning material customized to their notes. Recall and precision of the two advisors were evaluated by physician review. Students were given pre and post multiple choice question tests broadly covering geriatrics.

RESULTS

Of 102 students approached, 66 students consented and enrolled. The system sent 393 email alerts to 54 students (82%), including 270 for AD and 123 for AMS. Precision was 100% for AD and 93% for AMS. Recall was 69% for AD and 100% for AMS. Students mentioned ADs for 43 patients, with all mentions occurring after first having received an AD reminder. Students accessed educational links 34 times from the 393 email alerts. There was no difference in pre (mean 62%) and post (mean 60%) test scores.

CONCLUSIONS

The system effectively identified two educational opportunities using NLP applied to clinical notes and demonstrated a small change in student behavior. Use of electronic advisors such as these may provide a scalable model to assess specific competency elements and deliver educational opportunities.

An automated clinical alert system for newly-diagnosed atrial fibrillation

OBJECTIVE

Clinical decision support systems that notify providers of abnormal test results have produced mixed results. We sought to develop, implement, and evaluate the impact of a computer-based clinical alert system intended to improve atrial fibrillation stroke prophylaxis, and identify reasons providers do not implement a guideline-concordant response.

MATERIALS AND METHODS

We conducted a cohort study with historical controls among patients at a tertiary care hospital. We developed a decision rule to identify newly-diagnosed atrial fibrillation, automatically notify providers, and direct them to online evidence-based management guidelines. We tracked all notifications from December 2009 to February 2010 (notification period) and applied the same decision rule to all patients from December 2008 to February 2009 (control period). Primary outcomes were accuracy of notification (confirmed through chart review) and prescription of warfarin within 30 days.

RESULTS

During the notification period 604 notifications were triggered, of which 268 (44%) were confirmed as newly-diagnosed atrial fibrillation. The notifications not confirmed as newly-diagnosed involved patients with no recent electrocardiogram at our institution. Thirty-four of 125 high-risk patients (27%) received warfarin in the notification period, compared with 34 of 94 (36%) in the control period (odds ratio, 0.66 [95% CI, 0.37-1.17]; p = 0.16). Common reasons to not prescribe warfarin (identified from chart review of 151 patients) included upcoming surgical procedure, choice to use aspirin, and discrepancy between clinical notes and the medication record.

CONCLUSIONS

An automated system to identify newly-diagnosed atrial fibrillation, notify providers, and encourage access to management guidelines did not change provider behaviors.

Infobutton usage in Patient Portal MyHealth

Infobuttons have proven to be an important element in modern electronic health records (EHR), providing educational materials to both providers and patients. However, the usage of infobuttons in personalized health records (PHR) is only lightly documented in the literature. Patient-facing infobuttons pose a new challenge because patients have different questions and educational levels than professional users in EHRs. In this paper, we present usage data for patient-facing infobuttons that have recently been integrated in Intermountain Healthcare's patient portal MyHealth. We summarize use patterns by usage classified in modules, electronic resources (eResource), and infobutton sessions. Based on the analysis, we propose further enhancements to the current implementation of infobuttons in MyHealth.

A comprehensive information technology system to support physician learning at the point of care

MayoExpert is a multifaceted information system integrated with the electronic medical record (EMR) across Mayo Clinic's multisite health system. It was developed as a technology-based solution to manage information, standardize clinical practice, and promote and document learning in clinical contexts. Features include urgent test result notifications; models illustrating expert-approved care processes; concise, expert-approved answers to frequently asked questions (FAQs); a directory of topic-specific experts; and a portfolio for provider licensure and credentialing. The authors evaluate MayoExpert's reach, effectiveness, adoption, implementation, and maintenance. Evaluation data sources included usage statistics, user surveys, and pilot studies.As of October 2013, MayoExpert was available at 94 clinical sites in 12 states and contained 1,368 clinical topics, answers to 7,640 FAQs, and 92 care process models. In 2012, MayoExpert was accessed at least once by 2,578/3,643 (71%) staff physicians, 900/1,374 (66%) midlevel providers, and 1,728/2,291 (75%) residents and fellows. In a 2013 survey of MayoExpert users with 536 respondents, all features were highly rated (≥67% favorable). More providers reported using MayoExpert to answer questions before/after than during patient visits (68% versus 36%). During November 2012 to April 2013, MayoExpert sent 1,660 notifications of new-onset atrial fibrillation and 1,590 notifications of prolonged QT. MayoExpert has become part of routine clinical and educational operations, and its care process models now define Mayo Clinic best practices. MayoExpert's infrastructure and content will continue to expand with improved templates and content organization, new care process models, additional notifications, better EMR integration, and improved support for credentialing activities.

A Template for Authoring and Adapting Genomic Medicine Content in the eMERGE Infobutton Project

The Electronic Medical Records and Genomics (eMERGE) Network is a national consortium that is developing methods and best practices for using the electronic health record (EHR) for genomic medicine and research. We conducted a multi-site survey of information resources to support integration of pharmacogenomics into clinical care. This work aimed to: (a) characterize the diversity of information resource implementation strategies among eMERGE institutions; (b) develop a master template containing content topics of important for genomic medicine (as identified by the DISCERN-Genetics tool); and (c) assess the coverage of content topics among information resources developed by eMERGE institutions. Given that a standard implementation does not exist and sites relied on a diversity of information resources, we identified a need for a national effort to efficiently produce sharable genomic medicine resources capable of being accessed from the EHR. We discuss future areas of work to prepare institutions to use infobuttons for distributing standardized genomic content.

Enabling locally-developed content for access through the infobutton by means of automated concept annotation

Infobuttons have proven to be an increasingly important resource in providing a standardized approach to integrating useful educational materials at the point of care in electronic health records (EHRs). They provide a simple, uniform pathway for both patients and providers to receive pertinent education materials in a quick fashion from within EHRs and Personalized Health Records (PHRs). In recent years, the international standards organization Health Level Seven has balloted and approved a standards-based pathway for requesting and receiving data for infobuttons, simplifying some of the barriers for their adoption in electronic medical records and amongst content providers. Local content, developed by the hosting organization themselves, still needs to be indexed and annotated with appropriate metadata and terminologies in order to be fully accessible via the infobutton. In this manuscript we present an approach for automating the annotation of internally-developed patient education sheets with standardized terminologies and compare and contrast the approach with manual approaches used previously. We anticipate that a combination of system-generated and human reviewed annotations will provide the most comprehensive and effective indexing strategy, thereby allowing best access to internally-created content via the infobutton.

Evaluation of the effect of decision support on the efficiency of primary care providers in the outpatient practice

Background: Clinical decision support (CDS) for primary care has been shown to improve delivery of preventive services. However, there is little evidence for efficiency of physicians due to CDS assistance. In this article, we report a pilot study for measuring the impact of CDS on the time spent by physicians for deciding on preventive services and chronic disease management. Methods: We randomly selected 30 patients from a primary care practice, and assigned them to 10 physicians. The physicians were requested to perform chart review to decide on preventive services and chronic disease management for the assigned patients. The patients assignment was done in a randomized crossover design, such that each patient received 2 sets of recommendations—one from a physician with CDS assistance and the other from a different physician without CDS assistance. We compared the physician recommendations made using CDS assistance, with the recommendations made without CDS assistance. Results: The physicians required an average of 1 minute 44 seconds, when they were they had access to the decision support system and 5 minutes when they were unassisted. Hence the CDS assistance resulted in an estimated saving of 3 minutes 16 seconds (65%) of the physicians’ time, which was statistically significant (P < .0001). There was no statistically significant difference in the number of recommendations. Conclusion: Our findings suggest that CDS assistance significantly reduced the time spent by physicians for deciding on preventive services and chronic disease management. The result needs to be confirmed by performing similar studies at other institutions.

Determining primary care physician information needs to inform ambulatory visit note display

BACKGROUND

With the increase in the adoption of electronic health records (EHR) across the US, primary care physicians are experiencing information overload. The purpose of this pilot study was to determine the information needs of primary care physicians (PCPs) as they review clinic visit notes to inform EHR display.

METHOD

Data collection was conducted with 15 primary care physicians during semi-structured interviews, including a third party observer to control bias. Physicians reviewed major sections of an artificial but typical acute and chronic care visit note to identify the note sections that were relevant to their information needs. Statistical methods used were McNemar-Mosteller's and Cochran Q.

RESULTS

Physicians identified History of Present Illness (HPI), Assessment, and Plan (A&P) as the most important sections of a visit note. In contrast, they largely judged the Review of Systems (ROS) to be superfluous. There was also a statistical difference in physicians' highlighting among all seven major note sections in acute (p = 0.00) and chronic (p = 0.00) care visit notes.

CONCLUSION

A&P and HPI sections were most frequently identified as important which suggests that physicians may have to identify a few key sections out of a long, unnecessarily verbose visit note. ROS is viewed by doctors as mostly "not needed," but can have relevant information. The ROS can contain information needed for patient care when other sections of the Visit note, such as the HPI, lack the relevant information. Future studies should include producing a display that provides only relevant information to increase physician efficiency at the point of care. Also, research on moving A&P to the top of visit notes instead of having A&P at the bottom of the page is needed, since those are usually the first sections physicians refer to and reviewing from top to bottom may cause cognitive load.

Information needs and information-seeking behaviour analysis of primary care physicians and nurses: a literature review

BACKGROUND

The increase in the adoption of electronic health records (EHR) has contributed to physicians and nurses experiencing information overload. To address the problem of information overload, an assessment of the information needs of physicians and nurses will assist in understanding what they view as useful information to make patient care more efficient.

OBJECTIVE

To analyse studies that assessed the information needs and information-seeking behaviour of physicians and nurses in a primary care setting to develop a better understanding of what information to present to physicians when they making clinical decisions.

METHOD

A literature review of studies was conducted with a comprehensive search in PubMed, cinahl, scopus, as well as examination of references from relevant papers and hand-searched articles to identify articles applicable to this review.

RESULTS

Of the papers reviewed the most common information needs found among physicians and nurses were related to diagnoses, drug(s) and treatment/therapy. Colleagues remain a preferred information source among physicians and nurses; however, a rise in Internet usage is apparent.

CONCLUSION

Physicians and nurses need access to the Internet and job-specific resources to find practitioner-oriented information. In addition, effective usage of resources is important for improving patient care.

Use of health information technology to reduce diagnostic errors

BACKGROUND

Health information technology (HIT) systems have the potential to reduce delayed, missed or incorrect diagnoses. We describe and classify the current state of diagnostic HIT and identify future research directions.

METHODS

A multi-pronged literature search was conducted using PubMed, Web of Science, backwards and forwards reference searches and contributions from domain experts. We included HIT systems evaluated in clinical and experimental settings as well as previous reviews, and excluded radiology computer-aided diagnosis, monitor alerts and alarms, and studies focused on disease staging and prognosis. Articles were organised within a conceptual framework of the diagnostic process and areas requiring further investigation were identified.

RESULTS

HIT approaches, tools and algorithms were identified and organised into 10 categories related to those assisting: (1) information gathering; (2) information organisation and display; (3) differential diagnosis generation; (4) weighing of diagnoses; (5) generation of diagnostic plan; (6) access to diagnostic reference information; (7) facilitating follow-up; (8) screening for early detection in asymptomatic patients; (9) collaborative diagnosis; and (10) facilitating diagnostic feedback to clinicians. We found many studies characterising potential interventions, but relatively few evaluating the interventions in actual clinical settings and even fewer demonstrating clinical impact.

CONCLUSIONS

Diagnostic HIT research is still in its early stages with few demonstrations of measurable clinical impact. Future efforts need to focus on: (1) improving methods and criteria for measurement of the diagnostic process using electronic data; (2) better usability and interfaces in electronic health records; (3) more meaningful incorporation of evidence-based diagnostic protocols within clinical workflows; and (4) systematic feedback of diagnostic performance.

Terminology challenges implementing the HL7 context-aware knowledge retrieval ('Infobutton') standard

Point-of-care information needs are common and frequently unmet. One solution to this problem is the use of Infobuttons, which are context-sensitive links from electronic health records (EHR) to knowledge resources, sometimes involving an intermediate broker known as an Infobutton Manager. Health Level Seven (HL7) has developed the Context-Aware Knowledge Retrieval (Infobutton) standard to standardize the integration between EHR systems and knowledge resources. While the standard specifies a set of context attributes and standard terminologies, it leaves to knowledge resources the flexibility to decide how to use these attributes and terminologies to retrieve the most relevant content. This paper describes some of the challenges faced by knowledge resources in trying to locate the most relevant content based on the attribute values for a given Infobutton request. Various approaches to content retrieval are discussed, including the role of indexing with standardized codes, the role of text-based search engines together with their ranking algorithms, and the role of hybrid approaches. Knowledge resource developers must carefully consider business rules, heuristics, and precision/recall tradeoffs when implementing the HL7 Infobutton standard.

Disseminating context-specific access to online knowledge resources within electronic health record systems

Clinicians' patient care information needs are frequent and largely unmet. Online knowledge resources are available that can help clinicians meet these information needs. Yet, significant barriers limit the use of these resources within the clinical workflow. Infobuttons are clinical decision support tools that use the clinical context (e.g., institution, user, patient) within electronic health record (EHR) systems to anticipate clinicians' questions and provide automated links to relevant information in knowledge resources. This paper describes OpenInfobutton (www.openinfobutton.org): a standards-based, open source Web service that was designed to disseminate infobutton capabilities in multiple EHR systems and healthcare organizations. OpenInfobutton has been successfully integrated with 38 knowledge resources at 5 large healthcare organizations in the United States. We describe the OpenInfobutton architecture, knowledge resource integration, and experiences at five large healthcare organizations.

Evolution in clinical knowledge management strategy at Intermountain Healthcare

In this manuscript, we present an overview of the clinical knowledge management strategy at Intermountain Healthcare in support of our electronic medical record systems. Intermountain first initiated efforts in developing a centralized enterprise knowledge repository in 2001. Applications developed, areas of emphasis served, and key areas of focus are presented. We also detail historical and current areas of emphasis, in response to business needs.

Incorporating personalized gene sequence variants, molecular genetics knowledge, and health knowledge into an EHR prototype based on the Continuity of Care Record standard

OBJECTIVES

The current volume and complexity of genetic tests, and the molecular genetics knowledge and health knowledge related to interpretation of the results of those tests, are rapidly outstripping the ability of individual clinicians to recall, understand and convey to their patients information relevant to their care. The tailoring of molecular genetics knowledge and health knowledge in clinical settings is important both for the provision of personalized medicine and to reduce clinician information overload. In this paper we describe the incorporation, customization and demonstration of molecular genetic data (mainly sequence variants), molecular genetics knowledge and health knowledge into a standards-based electronic health record (EHR) prototype developed specifically for this study.

METHODS

We extended the CCR (Continuity of Care Record), an existing EHR standard for representing clinical data, to include molecular genetic data. An EHR prototype was built based on the extended CCR and designed to display relevant molecular genetics knowledge and health knowledge from an existing knowledge base for cystic fibrosis (OntoKBCF). We reconstructed test records from published case reports and represented them in the CCR schema. We then used the EHR to dynamically filter molecular genetics knowledge and health knowledge from OntoKBCF using molecular genetic data and clinical data from the test cases.

RESULTS

The molecular genetic data were successfully incorporated in the CCR by creating a category of laboratory results called "Molecular Genetics" and specifying a particular class of test ("Gene Mutation Test") in this category. Unlike other laboratory tests reported in the CCR, results of tests in this class required additional attributes ("Molecular Structure" and "Molecular Position") to support interpretation by clinicians. These results, along with clinical data (age, sex, ethnicity, diagnostic procedures, and therapies) were used by the EHR to filter and present molecular genetics knowledge and health knowledge from OntoKBCF.

CONCLUSIONS

This research shows a feasible model for delivering patient sequence variants and presenting tailored molecular genetics knowledge and health knowledge via a standards-based EHR system prototype. EHR standards can be extended to include the necessary patient data (as we have demonstrated in the case of the CCR), while knowledge can be obtained from external knowledge bases that are created and maintained independently from the EHR. This approach can form the basis for a personalized medicine framework, a more comprehensive standards-based EHR system and a potential platform for advancing translational research by both disseminating results and providing opportunities for new insights into phenotype-genotype relationships.

Implementations of the HL7 Context-Aware Knowledge Retrieval ("Infobutton") Standard: challenges, strengths, limitations, and uptake

To support clinical decision-making, computerized information retrieval tools known as "infobuttons" deliver contextually-relevant knowledge resources into clinical information systems. The Health Level Seven International (HL7) Context-Aware Knowledge Retrieval (Infobutton) Standard specifies a standard mechanism to enable infobuttons on a large scale.

OBJECTIVE

To examine the experience of organizations in the course of implementing the HL7 Infobutton Standard.

METHOD

Cross-sectional online survey and in-depth phone interviews.

RESULTS

A total of 17 organizations participated in the study. Analysis of the in-depth interviews revealed 20 recurrent themes. Implementers underscored the benefits, simplicity, and flexibility of the HL7 Infobutton Standard. Yet, participants voiced the need for easier access to standard specifications and improved guidance to beginners. Implementers predicted that the Infobutton Standard will be widely or at least fairly well adopted in the next 5 years, but uptake will depend largely on adoption among electronic health record (EHR) vendors. To accelerate EHR adoption of the Infobutton Standard, implementers recommended HL7-compliant infobutton capabilities to be included in the United States Meaningful Use Certification Criteria for EHR systems.

LIMITATIONS

Opinions and predictions should be interpreted with caution, since all the participant organizations have successfully implemented the standard and over half of the organizations were actively engaged in the development of the standard.

CONCLUSION

Overall, implementers reported a very positive experience with the HL7 Infobutton Standard. Despite indications of increasing uptake, measures should be taken to stimulate adoption of the Infobutton Standard among EHR vendors. Widespread adoption of the Infobutton Standard has the potential to bring contextually relevant clinical decision support content into the healthcare provider workflow.

Impact of knowledge resources linked to an electronic health record on frequency of unnecessary tests and treatments

INTRODUCTION

Electronic knowledge resources have the potential to rapidly provide answers to clinicians' questions. We sought to determine clinicians' reasons for searching these resources, the rate of finding relevant information, and the perceived clinical impact of the information they retrieved.

METHODS

We asked general internists, family physicians, and clinical nurse practitioners to complete the Information Assessment Method (IAM) survey after searching 1 of 2 electronic knowledge resources linked in the electronic health record. IAM stimulates reflection on the relevance, cognitive impact, use, and potential health outcomes of retrieved clinical information.

RESULTS

Forty-two clinicians rated 502 searches (mean 12, range 1-48) and reported finding information 75% (n = 375) of the time. The most common reasons for searching were to address a clinical question (411, 82%) and for curiosity (75, 15%). In 68% of the rated searches (341), participants indicated they would use the retrieved information for at least 1 patient. In 31% (157) of rated searches, clinicians expected the retrieved information to benefit the patient by avoiding an unnecessary or inappropriate treatment, diagnostic procedure, or preventive intervention.

CONCLUSIONS

Searches in electronic knowledge resources frequently yield relevant information that may benefit the patient by, for example, avoiding an inappropriate diagnostic procedure or treatment. Knowing that searches for answers to clinical questions can result in patient health benefits should intensify efforts to encourage clinicians to pursue answers to their questions.

Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems

BACKGROUND

Clinical decision support (CDS) is a valuable tool for improving healthcare quality and lowering costs. However, there is no comprehensive taxonomy of types of CDS and there has been limited research on the availability of various CDS tools across current electronic health record (EHR) systems.

OBJECTIVE

To develop and validate a taxonomy of front-end CDS tools and to assess support for these tools in major commercial and internally developed EHRs.

STUDY DESIGN AND METHODS

We used a modified Delphi approach with a panel of 11 decision support experts to develop a taxonomy of 53 front-end CDS tools. Based on this taxonomy, a survey on CDS tools was sent to a purposive sample of commercial EHR vendors (n=9) and leading healthcare institutions with internally developed state-of-the-art EHRs (n=4).

RESULTS

Responses were received from all healthcare institutions and 7 of 9 EHR vendors (response rate: 85%). All 53 types of CDS tools identified in the taxonomy were found in at least one surveyed EHR system, but only 8 functions were present in all EHRs. Medication dosing support and order facilitators were the most commonly available classes of decision support, while expert systems (eg, diagnostic decision support, ventilator management suggestions) were the least common.

CONCLUSION

We developed and validated a comprehensive taxonomy of front-end CDS tools. A subsequent survey of commercial EHR vendors and leading healthcare institutions revealed a small core set of common CDS tools, but identified significant variability in the remainder of clinical decision support content.

AskHERMES: An online question answering system for complex clinical questions

OBJECTIVE

Clinical questions are often long and complex and take many forms. We have built a clinical question answering system named AskHERMES to perform robust semantic analysis on complex clinical questions and output question-focused extractive summaries as answers.

DESIGN

This paper describes the system architecture and a preliminary evaluation of AskHERMES, which implements innovative approaches in question analysis, summarization, and answer presentation. Five types of resources were indexed in this system: MEDLINE abstracts, PubMed Central full-text articles, eMedicine documents, clinical guidelines and Wikipedia articles.

MEASUREMENT

We compared the AskHERMES system with Google (Google and Google Scholar) and UpToDate and asked physicians to score the three systems by ease of use, quality of answer, time spent, and overall performance.

RESULTS

AskHERMES allows physicians to enter a question in a natural way with minimal query formulation and allows physicians to efficiently navigate among all the answer sentences to quickly meet their information needs. In contrast, physicians need to formulate queries to search for information in Google and UpToDate. The development of the AskHERMES system is still at an early stage, and the knowledge resource is limited compared with Google or UpToDate. Nevertheless, the evaluation results show that AskHERMES' performance is comparable to the other systems. In particular, when answering complex clinical questions, it demonstrates the potential to outperform both Google and UpToDate systems.

CONCLUSIONS

AskHERMES, available at http://www.AskHERMES.org, has the potential to help physicians practice evidence-based medicine and improve the quality of patient care.

Connecting knowledge resources to the veterinary electronic health record: opportunities for learning at point of care

Electronic health records (EHRs) provide clinical learning opportunities through quick and contextual linkage of patient signalment, symptom, and diagnosis data with knowledge resources covering tests, drugs, conditions, procedures, and client instructions. This paper introduces the EHR standards for linkage and the partners-practitioners, content publishers, and software developers-necessary to leverage this possibility in veterinary medicine. The efforts of the American Animal Hospital Association (AAHA) Electronic Health Records Task Force to partner with veterinary practice management systems to improve the use of controlled vocabulary is a first step in the development of standards for sharing knowledge at the point of care. The Veterinary Medical Libraries Section (VMLS) of the Medical Library Association's Task Force on Connecting the Veterinary Health Record to Information Resources compiled a list of resources of potential use at point of care. Resource details were drawn from product Web sites and organized by a metric used to evaluate medical point-of-care resources. Additional information was gathered from questions sent by e-mail and follow-up interviews with two practitioners, a hospital network, two software developers, and three publishers. Veterinarians with electronic records use a variety of information resources that are not linked to their software. Systems lack the infrastructure to use the Infobutton standard that has been gaining popularity in human EHRs. While some veterinary knowledge resources are digital, publisher sites and responses do not indicate a Web-based linkage of veterinary resources with EHRs. In order to facilitate lifelong learning and evidence-based practice, veterinarians and educators of future practitioners must demonstrate to veterinary practice software developers and publishers a clinically-based need to connect knowledge resources to veterinary EHRs.

Evaluation of email alerts in practice: Part 1. Review of the literature on clinical emailing channels

RATIONALE

Methods to systematically assess electronic knowledge resources by health professionals may enhance evaluation of these resources, knowledge exchange between information users and providers, and continuing professional development. We developed the Information Assessment Method (IAM) to document health professional perspectives on the relevance, cognitive impact, potential use and expected health outcomes of information delivered by (push) or retrieved from (pull) electronic knowledge resources. However, little is known about push communication in health sciences, and what we propose to call clinical emailing channels (CECs). CECs can be understood as a communication infrastructure that channels clinically relevant research knowledge, email alerts, from information providers to the inboxes of individual practitioners.

AIMS

In two companion papers, our objectives are to (part 1) explore CEC evaluation in routine practice, and (part 2) examine the content validity of the cognitive component of IAM.

METHODS

The present paper (part 1) critically reviews the literature in health sciences and four disciplines: communication, information studies, education and knowledge translation. Our review addresses the following questions. What are CECs? How are they assessed?

RESULTS

The review contributes to better define CECs, and proposes a 'push-pull-acquisition-cognition-application' evaluation framework, which is operationalized by IAM.

CONCLUSION

Compared with existing evaluation tools, our review suggests IAM is comprehensive, generic and systematic.

Metadata extraction routines for improving infobutton performance

Infobuttons have been proven as an effective means for providing quick, context-specific links to pertinent information resources at the point of care. Current infobutton manager implementations, however, lack the ability to exchange metadata, are limited to a relatively small set of information providers, and are targeted primarily for a clinician audience. As part of a local effort to implement infobuttons for patient use via a tethered personal health record, we present a series of metadata extraction routines. These routines were constructed to extract key pieces of information from health information providers on the Internet, including content coverage, language availability, and readability scores. The extraction routines were tested using thirty different disease conditions against eight different providers. The routines yielded 183 potential infobutton targets and associated metadata for each. The capabilities of the extraction routines will be expanded to cover new types of metadata in the future.

[The German Medical eLibrary]

The internet offers doctors plenty of high-value information for medical questions. However, several barriers impair the preoccupation with this information, most notably lack of time, far too much offered information and missing confidence in the reliability of the offers. The following article points out how the Agency for Quality in Medicine developed the Internet portal "German Medical eLibrary" (Arztbibliothek) in which information is transparently pre-selected and grouped in a structured way as well as easy to find with the help of a simple, user-friendly search function. Evidence-based high-quality information is available in an easy, quick and well-directed manner. Thus the "German Medical eLibrary" facilitates knowledge management as a cornerstone of high-quality medical care.

Clinician search behaviors may be influenced by search engine design

Background

Searching the Web for documents using information retrieval systems plays an important part in clinicians’ practice of evidence-based medicine. While much research focuses on the design of methods to retrieve documents, there has been little examination of the way different search engine capabilities influence clinician search behaviors.

Objectives

Previous studies have shown that use of task-based search engines allows for faster searches with no loss of decision accuracy compared with resource-based engines. We hypothesized that changes in search behaviors may explain these differences.

Methods

In all, 75 clinicians (44 doctors and 31 clinical nurse consultants) were randomized to use either a resource-based or a task-based version of a clinical information retrieval system to answer questions about 8 clinical scenarios in a controlled setting in a university computer laboratory. Clinicians using the resource-based system could select 1 of 6 resources, such as PubMed; clinicians using the task-based system could select 1 of 6 clinical tasks, such as diagnosis. Clinicians in both systems could reformulate search queries. System logs unobtrusively capturing clinicians’ interactions with the systems were coded and analyzed for clinicians’ search actions and query reformulation strategies.

Results

The most frequent search action of clinicians using the resource-based system was to explore a new resource with the same query, that is, these clinicians exhibited a “breadth-first” search behaviour. Of 1398 search actions, clinicians using the resource-based system conducted 401 (28.7%, 95% confidence interval [CI] 26.37-31.11) in this way. In contrast, the majority of clinicians using the task-based system exhibited a “depth-first” search behavior in which they reformulated query keywords while keeping to the same task profiles. Of 585 search actions conducted by clinicians using the task-based system, 379 (64.8%, 95% CI 60.83-68.55) were conducted in this way.

Conclusions

This study provides evidence that different search engine designs are associated with different user search behaviors.

Pilot study of linking Web-based supplemental interpretive information to laboratory test reports

Electronic medical records have the ability to link to reference material, providing clinicians with immediate access to information relevant to patient care. Adding relevant links to laboratory test results could add value while minimizing the volume of ancillary text presented. We provided Web-based universal resource locator (URL) links with all results of 7 laboratory tests ordered at ARUP Laboratories (Salt Lake City, UT). URL links provided were modified 7 months later, and use between initial and subsequent URLs was tracked to establish frequency and duration of access to supplemental Web information. Monthly Web-site hit rates for individual tests varied from 0.00% to 3.00% (median, 0.12%). Rare and specialty tests averaged higher hit rates. There was no decay in hit rate 9 months after URLs were removed from test reports. We conclude that links to reference material are accessed by clinicians. The use of Web links months after links were no longer published raises an important issue of long-term maintenance and the resources required to support these features.