Early Intervention Referral Information, Transmission, and Sources-A Survey of State Part C Coordinators and Analysis of Referral Forms

OBJECTIVE

Early Intervention (EI) referral is a key connector between health care and early childhood systems serving children with developmental risks. This study aimed to describe the US network of EI referrals by answering the following: "What information is sent to EI?", "Who sends it?", and "How is it sent?"

METHOD

This study combined an analysis of national document-based and website-based referral forms with a survey of state Part C Coordinators (PCCs). Data on referral forms were systematically collected from state agency websites. PCCs from 52 jurisdictions were surveyed to assess current EI referral practices. Descriptive statistics were used for responses to multiple-choice items; free-text answers were condensed into key study themes.

RESULTS

EI referral forms came as e-documents (81%) or websites (35%), and 72% were in English alone. They emphasized family and referral source contact information and reason for the referral. The survey results indicated that health care (45%) sends the most referrals, followed by families (30%). EI agencies received referrals by phone (38%), electronically (23%), e-mail (17%), and fax (17%), and PCCs valued this diversity of methods. Few states received referral data directly from electronic health records (EHRs); however, PCCs hope to eventually receive referrals through websites, mobile devices, and EHRs.

CONCLUSION

EI referral data flow is complex, with opportunities for loss of children to follow-up. This study describes how EI referrals occur and provides examples of how communication and access to information may be improved.

Exploiting the power of information in medical education

The explosion of medical information demands a thorough reconsideration of medical education, including what we teach and assess, how we educate, and whom we educate. Physicians of the future will need to be self-aware, self-directed, resource-effective team players who can synthesize and apply summarized information and communicate clearly. Training in metacognition, data science, informatics, and artificial intelligence is needed. Education programs must shift focus from content delivery to providing students explicit scaffolding for future learning, such as the Master Adaptive Learner model. Additionally, educators should leverage informatics to improve the process of education and foster individualized, precision education. Finally, attributes of the successful physician of the future should inform adjustments in recruitment and admissions processes. This paper explores how member schools of the American Medical Association Accelerating Change in Medical Education Consortium adjusted all aspects of educational programming in acknowledgment of the rapid expansion of information.

Assessing the Transition of Training in Health Systems Science From Undergraduate to Graduate Medical Education

BACKGROUND

The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation.

OBJECTIVE

This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME).

METHODS

ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models.

RESULTS

At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups.

CONCLUSIONS

Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

Priorities of Hybrid Clinician-Managers: A Qualitative Study of How Managers Balance Clinical Quality Among Competing Responsibilities

OBJECTIVE

Oversight of clinical quality is only one of physical therapy managers' multiple responsibilities. With the move to value-based care, organizations need sound management to navigate this evolving reimbursement landscape. Previous research has not explored how competing priorities affect physical therapy managers' oversight of clinical quality. The purpose of this study was to create a preliminary model of the competing priorities, motivations, and responsibilities of managers while overseeing clinical quality.

METHODS

This qualitative study used the Rapid Qualitative Inquiry method. A purposive sample of 40 physical therapy managers and corporate leaders was recruited. A research team performed semi-structured interviews and observations in outpatient practices. The team used a grounded theory-based immersion/crystallization analysis approach. Identified themes delineated the competing priorities and workflows these managers use in their administrative duties.

RESULTS

Six primary themes were identified that illustrate how managers: (1) balance managerial and professional priorities; (2) are susceptible to stakeholder influences; (3) experience internal conflict; (4) struggle to measure and define quality objectively; (5) are influenced by the culture and structure of their respective organizations; and (6) have professional needs apart from the needs of their clinics.

CONCLUSION

Generally, managers' focus on clinical quality is notably less comprehensive than their focus on clinical operations. Additionally, the complex role of hybrid clinician-manager leaves limited time beyond direct patient care for administrative duties. Managers in organizations that hold them accountable to quality-based metrics have more systematic clinical quality oversight processes.

IMPACT

This study gives physical therapy organizations a framework of factors that can be influenced to better facilitate managers' effective oversight of clinical quality. Organizations offering support for those managerial responsibilities will be well positioned to thrive in the new fee-for-value care structure.

Registered Nurse Strain Detection Using Ambient Data: An Exploratory Study of Underutilized Operational Data Streams in the Hospital Workplace

BACKGROUND

Registered nurses (RNs) regularly adapt their work to ever-changing situations but routine adaptation transforms into RN strain when service demand exceeds staff capacity and patients are at risk of missed or delayed care. Dynamic monitoring of RN strain could identify when intervention is needed, but comprehensive views of RN work demands are not readily available. Electronic care delivery tools such as nurse call systems produce ambient data that illuminate workplace activity, but little is known about the ability of these data to predict RN strain.

OBJECTIVES

The purpose of this study was to assess the utility of ambient workplace data, defined as time-stamped transaction records and log file data produced by non-electronic health record care delivery tools (e.g., nurse call systems, communication devices), as an information channel for automated sensing of RN strain.

METHODS

In this exploratory retrospective study, ambient data for a 1-year time period were exported from electronic nurse call, medication dispensing, time and attendance, and staff communication systems. Feature sets were derived from these data for supervised machine learning models that classified work shifts by unplanned overtime. Models for three timeframes -8, 10, and 12 hours-were created to assess each model's ability to predict unplanned overtime at various points across the work shift.

RESULTS

Classification accuracy ranged from 57 to 64% across three analysis timeframes. Accuracy was lowest at 10 hours and highest at shift end. Features with the highest importance include minutes spent using a communication device and percent of medications delivered via a syringe.

CONCLUSION

Ambient data streams can serve as information channels that contain signals related to unplanned overtime as a proxy indicator of RN strain as early as 8 hours into a work shift. This study represents an initial step toward enhanced detection of RN strain and proactive prevention of missed or delayed patient care.

Subtle cues: Qualitative elicitation of signs of capacity strain in the hospital workplace

Through everyday care experiences, nurses develop expertise in recognition of capacity strain in hospital workplaces. Through qualitative interview, experienced nurses identify common activity changes and adaptive work strategies that may signal an imbalance between patient demand and service supply at the bedside. Activity change examples include nurse helping behaviors across patient assignments, increased volume of nurse calls from patient rooms, and decreased presence of staff at the nurses' station. Adaptive work strategies encompass actions taken to recruit resources, move work in time, reduce work demands, or reduce thoroughness of task performance. Nurses' knowledge of perceptible signs of strain provides a foundation for future exploration and development of real-time indicators of capacity strain in hospital-based work systems.

Modeling variation of clinical team processes with multiple sequence alignment

Our objective was to model process variation of Emergency Medical Service teams responding to simulated pediatric emergencies and determine if sequence alignment distinguishes performance quality. We performed a retrospective process analysis by watching and coding activities in videos from standardized simulations of 42 Emergency Medical Service teams. Teams were classified into high- or low-performing groups based on the Clinical Teamwork Scale™. Activities were coded according to resuscitation tasks, performer, and times. We used ClustalG to align task sequences within and between groups, and measured similarity. Teams within and between performance levels had an average sequence similarity of 52 ± 7% and 50 ± 7%. Teams performed clinically appropriate tasks that varied in prioritization, for example, performing compressions or connecting the EKG monitor early. There was no statistical difference in gross similarity between groups but specific differences in prioritization may have had clinically meaningful implications. Alignment could improve by accounting for task duration and concurrency.

Clinical decisions support malfunctions in a commercial electronic health record

OBJECTIVES

Determine if clinical decision support (CDS) malfunctions occur in a commercial electronic health record (EHR) system, characterize their pathways and describe methods of detection.

METHODS

We retrospectively examined the firing rate for 226 alert type CDS rules for detection of anomalies using both expert visualization and statistical process control (SPC) methods over a five year period. Candidate anomalies were investigated and validated.

RESULTS

Twenty-one candidate CDS anomalies were identified from 8,300 alert-months. Of these candidate anomalies, four were confirmed as CDS malfunctions, eight as false-positives, and nine could not be classified. The four CDS malfunctions were a result of errors in knowledge management: 1) inadvertent addition and removal of a medication code to the electronic formulary list; 2) a seasonal alert which was not activated; 3) a change in the base data structures; and 4) direct editing of an alert related to its medications. 154 CDS rules (68%) were amenable to SPC methods and the test characteristics were calculated as a sensitivity of 95%, positive predictive value of 29% and F-measure 0.44.

DISCUSSION

CDS malfunctions were found to occur in our EHR. All of the pathways for these malfunctions can be described as knowledge management errors. Expert visualization is a robust method of detection, but is resource intensive. SPC-based methods, when applicable, perform reasonably well retrospectively.

CONCLUSION

CDS anomalies were found to occur in a commercial EHR and visual detection along with SPC analysis represents promising methods of malfunction detection.

ACC/AHA Special Report: Clinical Practice Guideline Implementation Strategies: A Summary of Systematic Reviews by the NHLBI Implementation Science Work Group: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines

BACKGROUND

In 2008, the National Heart, Lung, and Blood Institute convened an Implementation Science Work Group to assess evidence-based strategies for effectively implementing clinical practice guidelines. This was part of a larger effort to update existing clinical practice guidelines on cholesterol, blood pressure, and overweight/obesity.

OBJECTIVES

Review evidence from the published implementation science literature and identify effective or promising strategies to enhance the adoption and implementation of clinical practice guidelines.

METHODS

This systematic review was conducted on 4 critical questions, each focusing on the adoption and effectiveness of 4 intervention strategies: (1) reminders, (2) educational outreach visits, (3) audit and feedback, and (4) provider incentives. A scoping review of the Rx for Change database of systematic reviews was used to identify promising guideline implementation interventions aimed at providers. Inclusion and exclusion criteria were developed a priori for each question, and the published literature was initially searched up to 2012, and then updated with a supplemental search to 2015. Two independent reviewers screened the returned citations to identify relevant reviews and rated the quality of each included review.

RESULTS

Audit and feedback and educational outreach visits were generally effective in improving both process of care (15 of 21 reviews and 12 of 13 reviews, respectively) and clinical outcomes (7 of 12 reviews and 3 of 5 reviews, respectively). Provider incentives showed mixed effectiveness for improving both process of care (3 of 4 reviews) and clinical outcomes (3 reviews equally distributed between generally effective, mixed, and generally ineffective). Reminders showed mixed effectiveness for improving process of care outcomes (27 reviews with 11 mixed and 3 generally ineffective results) and were generally ineffective for clinical outcomes (18 reviews with 6 mixed and 9 generally ineffective results). Educational outreach visits (2 of 2 reviews), reminders (3 of 4 reviews), and provider incentives (1 of 1 review) were generally effective for cost reduction. Educational outreach visits (1 of 1 review) and provider incentives (1 of 1 review) were also generally effective for cost-effectiveness outcomes. Barriers to clinician adoption or adherence to guidelines included time constraints (8 reviews/overviews); limited staffing resources (2 overviews); timing (5 reviews/overviews); clinician skepticism (5 reviews/overviews); clinician knowledge of guidelines (4 reviews/overviews); and higher age of the clinician (1 overview). Facilitating factors included guideline characteristics such as format, resources, and end-user involvement (6 reviews/overviews); involving stakeholders (5 reviews/overviews); leadership support (5 reviews/overviews); scope of implementation (5 reviews/overviews); organizational culture such as multidisciplinary teams and low-baseline adherence (9 reviews/overviews); and electronic guidelines systems (3 reviews).

CONCLUSION

The strategies of audit and feedback and educational outreach visits were generally effective in improving both process of care and clinical outcomes. Reminders and provider incentives showed mixed effectiveness, or were generally ineffective. No general conclusion could be reached about cost effectiveness, because of limitations in the evidence. Important gaps exist in the evidence on effectiveness of implementation interventions, especially regarding clinical outcomes, cost effectiveness and contextual issues affecting successful implementation.

Barriers and facilitators to exchanging health information: a systematic review

OBJECTIVES

We conducted a systematic review of studies assessing facilitators and barriers to use of health information exchange (HIE).

METHODS

We searched MEDLINE, PsycINFO, CINAHL, and the Cochrane Library databases between January 1990 and February 2015 using terms related to HIE. English-language studies that identified barriers and facilitators of actual HIE were included. Data on study design, risk of bias, setting, geographic location, characteristics of the HIE, perceived barriers and facilitators to use were extracted and confirmed.

RESULTS

Ten cross-sectional, seven multiple-site case studies, and two before-after studies that included data from several sources (surveys, interviews, focus groups, and observations of users) evaluated perceived barriers and facilitators to HIE use. The most commonly cited barriers to HIE use were incomplete information, inefficient workflow, and reports that the exchanged information that did not meet the needs of users. The review identified several facilitators to use.

DISCUSSION

Incomplete patient information was consistently mentioned in the studies conducted in the US but not mentioned in the few studies conducted outside of the US that take a collective approach toward healthcare. Individual patients and practices in the US may exercise the right to participate (or not) in HIE which effects the completeness of patient information available to be exchanged. Workflow structure and user roles are key but understudied.

CONCLUSIONS

We identified several facilitators in the studies that showed promise in promoting electronic health data exchange: obtaining more complete patient information; thoughtful workflow that folds in HIE; and inclusion of users early in implementation.

A mobile/web app for long distance caregivers of older adults: functional requirements and design implications from a user centered design process

Recent trends of population aging and globalization have required an increasing number of individuals to act as long distance caregivers (LDCs) to aging family members. Information technology solutions may ease the burden placed on LDCs by providing remote monitoring, easier access to information and enhanced communication. While some technology tools have been introduced, the information and technology needs of LDCs in particular are not well understood. Consequently, a needs assessment was performed by using video conferencing software to conduct semi-structured interviews with 10 LDCs. Interviews were enriched through the use of stimulus materials that included the demonstration of a prototype LDC health management web/mobile app. Responses were recorded, transcribed and then analyzed. Subjects indicated that information regarding medication regimens and adherence, calendaring, and cognitive health were most needed. Participants also described needs for video calling, activity data regarding sleep and physical exercise, asynchronous communication, photo sharing, journaling, access to online health resources, real-time monitoring, an overall summary of health, and feedback/suggestions to help them improve as caregivers. In addition, all respondents estimated their usage of a LDC health management website would be at least once per week, with half indicating a desire to access the website from a smartphone. These findings are being used to inform the design of a LDC health management website to promote the meaningful involvement of distant family members in the care of older adults.

Beyond information retrieval and electronic health record use: competencies in clinical informatics for medical education

Physicians in the 21st century will increasingly interact in diverse ways with information systems, requiring competence in many aspects of clinical informatics. In recent years, many medical school curricula have added content in information retrieval (search) and basic use of the electronic health record. However, this omits the growing number of other ways that physicians are interacting with information that includes activities such as clinical decision support, quality measurement and improvement, personal health records, telemedicine, and personalized medicine. We describe a process whereby six faculty members representing different perspectives came together to define competencies in clinical informatics for a curriculum transformation process occurring at Oregon Health & Science University. From the broad competencies, we also developed specific learning objectives and milestones, an implementation schedule, and mapping to general competency domains. We present our work to encourage debate and refinement as well as facilitate evaluation in this area.

Clinical questions raised by clinicians at the point of care: a systematic review

IMPORTANCE

In making decisions about patient care, clinicians raise questions and are unable to pursue or find answers to most of them. Unanswered questions may lead to suboptimal patient care decisions.

OBJECTIVE

To systematically review studies that examined the questions clinicians raise in the context of patient care decision making.

DATA SOURCES

MEDLINE (from 1966), CINAHL (from 1982), and Scopus (from 1947), all through May 26, 2011. STUDY SELECTION Studies that examined questions raised and observed by clinicians (physicians, medical residents, physician assistants, nurse practitioners, nurses, dentists, and care managers) in the context of patient care were independently screened and abstracted by 2 investigators. Of 21,710 citations, 72 met the selection criteria.

DATA EXTRACTION AND SYNTHESIS

Question frequency was estimated by pooling data from studies with similar methods.

MAIN OUTCOMES AND MEASURES

Frequency of questions raised, pursued, and answered and questions by type according to a taxonomy of clinical questions. Thematic analysis of barriers to information seeking and the effects of information seeking on decision making. RESULTS In 11 studies, 7012 questions were elicited through short interviews with clinicians after each patient visit. The mean frequency of questions raised was 0.57 (95% CI, 0.38-0.77) per patient seen, and clinicians pursued 51% (36%-66%) of questions and found answers to 78% (67%-88%) of those they pursued. Overall, 34% of questions concerned drug treatment, and 24% concerned potential causes of a symptom, physical finding, or diagnostic test finding. Clinicians' lack of time and doubt that a useful answer exists were the main barriers to information seeking.

CONCLUSIONS AND RELEVANCE

Clinicians frequently raise questions about patient care in their practice. Although they are effective at finding answers to questions they pursue, roughly half of the questions are never pursued. This picture has been fairly stable over time despite the broad availability of online evidence resources that can answer these questions. Technology-based solutions should enable clinicians to track their questions and provide just-in-time access to high-quality evidence in the context of patient care decision making. Opportunities for improvement include the recent adoption of electronic health record systems and maintenance of certification requirements.

Comparison of antibiograms developed for inpatients and primary care outpatients

To support antimicrobial stewardship, some healthcare systems have begun creating outpatient antibiograms. We developed inpatient and primary care outpatient antibiograms for a regional health maintenance organization (HMO) and academic healthcare system (AHS). Antimicrobial susceptibilities from 16,428 Enterococcus, Staphylococcus aureus, Escherichia coli, Klebsiella pneumoniae, and Pseudomonas aeruginosa cultures from 2010 were summarized and compared. Methicillin susceptibility among S. aureus was similar in inpatients and primary care outpatients (HMO: 61.2% versus 61.9%, P = 0.951; AHS: 62.9% versus 63.3%, P > 0.999). E. coli susceptibility to trimethoprim/sulfamethoxazole was also similar (HMO: 81.8% versus 83.6%, P = 0.328; AHS: 77.2% versus 80.9%, P = 0.192), but ciprofloxacin susceptibility differed (HMO: 88.9% versus 94.6%, P < 0.001; AHS: 81.2% versus 90.6%, P < 0.001). In the HMO, ciprofloxacin-susceptible P. aeruginosa were more frequent in primary care outpatients than in inpatients (91.4% versus 79.0%, P = 0.007). Comparison of cumulative susceptibilities across settings yielded no consistent patterns; therefore, outpatient primary care antibiograms may more accurately inform prudent empiric antibiotic prescribing.

Perceptions of shared decision making and decision aids among rural primary care clinicians

BACKGROUND

Shared decision making (SDM) and decision aids (DAs) increase patients' involvement in health care decisions and enhance satisfaction with their choices. Studies of SDM and DAs have primarily occurred in academic centers and large health systems, but most primary care is delivered in smaller practices, and over 20% of Americans live in rural areas, where poverty, disease prevalence, and limited access to care may increase the need for SDM and DAs.

OBJECTIVE

To explore perceptions and practices of rural primary care clinicians regarding SDM and DAs.

DESIGN

Cross-sectional survey. Setting and Participants Primary care clinicians affiliated with the Oregon Rural Practice-based Research Network.

RESULTS

Surveys were returned by 181 of 231 eligible participants (78%); 174 could be analyzed. Two-thirds of participants were physicians, 84% practiced family medicine, and 55% were male. Sixty-five percent of respondents were unfamiliar with the term shared decision making, but following definition, 97% reported that they found the approach useful for conditions with multiple treatment options. Over 90% of clinicians perceived helping patients make decisions regarding chronic pain and health behavior change as moderate/hard in difficulty. Although 69% of respondents preferred that patients play an equal role in making decisions, they estimate that this happens only 35% of the time. Time was reported as the largest barrier to engaging in SDM (63%). Respondents were receptive to using DAs to facilitate SDM in print- (95%) or web-based formats (72%), and topic preference varied by clinician specialty and decision difficulty.

CONCLUSIONS

Rural clinicians recognized the value of SDM and were receptive to using DAs in multiple formats. Integration of DAs to facilitate SDM in routine patient care may require addressing practice operation and reimbursement.

Tiering drug-drug interaction alerts by severity increases compliance rates

OBJECTIVE

Few data exist measuring the effect of differentiating drug-drug interaction (DDI) alerts in computerized provider order entry systems (CPOE) by level of severity ("tiering"). We sought to determine if rates of provider compliance with DDI alerts in the inpatient setting differed when a tiered presentation was implemented.

DESIGN

We performed a retrospective analysis of alert log data on hospitalized patients at two academic medical centers during the period from 2/1/2004 through 2/1/2005. Both inpatient CPOE systems used the same DDI checking service, but one displayed alerts differentially by severity level (tiered presentation, including hard stops for the most severe alerts) while the other did not. Participants were adult inpatients who generated a DDI alert, and providers who wrote the orders. Alerts were presented during the order entry process, providing the clinician with the opportunity to change the patient's medication orders to avoid the interaction.

MEASUREMENTS

Rate of compliance to alerts at a tiered site compared to a non-tiered site.

RESULTS

We reviewed 71,350 alerts, of which 39,474 occurred at the non-tiered site and 31,876 at the tiered site. Compliance with DDI alerts was significantly higher at the site with tiered DDI alerts compared to the non-tiered site (29% vs. 10%, p < 0.001). At the tiered site, 100% of the most severe alerts were accepted, vs. only 34% at the non-tiered site; moderately severe alerts were also more likely to be accepted at the tiered site (29% vs. 10%).

CONCLUSION

Tiered alerting by severity was associated with higher compliance rates of DDI alerts in the inpatient setting, and lack of tiering was associated with a high override rate of more severe alerts.

Distributed cognition: An alternative model of cognition for medical informatics

BACKGROUND

Medical informatics has been guided by an individual-centered model of human cognition, inherited from classical theory of mind, in which knowledge, problem-solving, and information-processing responsible for intelligent behavior all derive from the inner workings of an individual agent.

OBJECTIVES AND RESULTS

In this paper we argue that medical informatics commitment to the classical model of cognition conflates the processing performed by the minds of individual agents with the processing performed by the larger distributed activity systems within which individuals operate. We review trends in cognitive science that seek to close the gap between general-purpose models of cognition and applied considerations of real-world human performance. One outcome is the theory of distributed cognition, in which the unit of analysis for understanding performance is the activity system which comprises a group of human actors, their tools and environment, and is organized by a particular history of goal-directed action and interaction.

CONCLUSION

We describe and argue for the relevance of distributed cognition to medical informatics, both for the study of human performance in healthcare and for the design of technologies meant to enhance this performance.

Advancing biomedical image retrieval: development and analysis of a test collection

OBJECTIVE

Develop and analyze results from an image retrieval test collection.

METHODS

After participating research groups obtained and assessed results from their systems in the image retrieval task of Cross-Language Evaluation Forum, we assessed the results for common themes and trends. In addition to overall performance, results were analyzed on the basis of topic categories (those most amenable to visual, textual, or mixed approaches) and run categories (those employing queries entered by automated or manual means as well as those using visual, textual, or mixed indexing and retrieval methods). We also assessed results on the different topics and compared the impact of duplicate relevance judgments.

RESULTS

A total of 13 research groups participated. Analysis was limited to the best run submitted by each group in each run category. The best results were obtained by systems that combined visual and textual methods. There was substantial variation in performance across topics. Systems employing textual methods were more resilient to visually oriented topics than those using visual methods were to textually oriented topics. The primary performance measure of mean average precision (MAP) was not necessarily associated with other measures, including those possibly more pertinent to real users, such as precision at 10 or 30 images.

CONCLUSIONS

We developed a test collection amenable to assessing visual and textual methods for image retrieval. Future work must focus on how varying topic and run types affect retrieval performance. Users' studies also are necessary to determine the best measures for evaluating the efficacy of image retrieval systems.

Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians

OBJECTIVE

Computerized physician documentation (CPD) has been implemented throughout the nation's Veterans Affairs Medical Centers (VAMCs) and is likely to increasingly replace handwritten documentation in other institutions. The use of this technology may affect educational and clinical activities, yet little has been reported in this regard. The authors conducted a qualitative study to determine the perceived impacts of CPD among faculty and housestaff in a VAMC.

DESIGN

A cross-sectional study was conducted using semistructured interviews with faculty (n = 10) and a group interview with residents (n = 10) at a VAMC teaching hospital.

MEASUREMENTS

Content analysis of field notes and taped transcripts were done by two independent reviewers using a grounded theory approach. Findings were validated using member checking and peer debriefing.

RESULTS

Four major themes were identified: (1) improved availability of documentation; (2) changes in work processes and communication; (3) alterations in document structure and content; and (4) mistakes, concerns, and decreased confidence in the data. With a few exceptions, subjects felt documentation was more available, with benefits for education and patient care. Other impacts of CPD were largely seen as detrimental to aspects of clinical practice and education, including documentation quality, workflow, professional communication, and patient care.

CONCLUSION

CPD is perceived to have substantial positive and negative impacts on clinical and educational activities and environments. Care should be taken when designing, implementing, and using such systems to avoid or minimize any harmful impacts. More research is needed to assess the extent of the impacts identified and to determine the best strategies to effectively deal with them.

Finding the answers in primary care: information seeking by rural and nonrural clinicians

BACKGROUND

Information systems for rural practice may assume that rural clinicians have different information seeking, but studies have not directly compared rural and nonrural information needs using common methodology.

OBJECTIVE

Compare rural and non-rural: 1) information needs; 2) information seeking; 3) effectiveness of information seeking; and 4) use of information resources.

DESIGN

Observation and interviews during one half-day of office practice; telephone follow-up 2-10 days later.

PARTICIPANTS & SETTING

Primary care physicians (39), nurse practitioners (42), and physician assistants (22) in ambulatory practices in rural and nonrural Oregon.

MEASURES

1) number of questions asked, 2) number of questions pursued, 3) number of questions answered), and 4) use of knowledge resources.

RESULTS

Rural clinicians practiced in smaller groups, but were otherwise similar to nonrural clinicians. During half-day interviews, clinicians cared for an average of 8.2 patients (95% CI 7.5 - 8.8) and asked an average of 0.83 questions per patient seen (95% CI 0.73 - 0.92). At follow up, they had pursued an average of 47% of their questions (95% CI 40 - 53%), and reported being successful in finding an answer to 77% of those they pursued (95% CI 70 - 84%). There were no statistically significant differences between rural and nonrural clinicians for any of these variables.

CONCLUSIONS

Rural and nonrural clinicians had similar information needs, information seeking, knowledge resource use, and effectiveness at finding answers to their questions. Human consultants, digital resources, and library-based resources were less available, but these differences in availability had little impact on their use.

Computerized physician order entry in U.S. hospitals: results of a 2002 survey

OBJECTIVE

To determine the availability of inpatient computerized physician order entry in U.S. hospitals and the degree to which physicians are using it.

DESIGN

Combined mail and telephone survey of 964 randomly selected hospitals, contrasting 2002 data and results of a survey conducted in 1997.

MEASUREMENTS

AVAILABILITY

computerized order entry has been installed and is available for use by physicians; inducement: the degree to which use of computers to enter orders is required of physicians; participation: the proportion of physicians at an institution who enter orders by computer; and saturation: the proportion of total orders at an institution entered by a physician using a computer.

RESULTS

The response rate was 65%. Computerized order entry was not available to physicians at 524 (83.7%) of 626 hospitals responding, whereas 60 (9.6%) reported complete availability and 41 (6.5%) reported partial availability. Of 91 hospitals providing data about inducement/requirement to use the system, it was optional at 31 (34.1%), encouraged at 18 (19.8%), and required at 42 (46.2%). At 36 hospitals (45.6%), more than 90% of physicians on staff use the system, whereas six (7.6%) reported 51-90% participation and 37 (46.8%) reported participation by fewer than half of physicians. Saturation was bimodal, with 25 (35%) hospitals reporting that more than 90% of all orders are entered by physicians using a computer and 20 (28.2%) reporting that less than 10% of all orders are entered this way.

CONCLUSION

Despite increasing consensus about the desirability of computerized physician order entry (CPOE) use, these data indicate that only 9.6% of U.S. hospitals presently have CPOE completely available. In those hospitals that have CPOE, its use is frequently required. In approximately half of those hospitals, more than 90% of physicians use CPOE; in one-third of them, more than 90% of orders are entered via CPOE.

Order creation and communication in healthcare

OBJECTIVES

The aim of this paper is to examine the adequacy of the concept of Physician Order Entry (POE) as a model for clinical systems, and to suggest an alternative understanding of the order creation and communication process.

METHODS

The study is based on an interpretative analysis of POE as a model for clinical systems and the results of our recent fieldwork.

RESULTS

Observations from our recent fieldwork suggest that orders, like patient care in general, emerge from interactions among patients, physicians, nurses, family members, and others, employing a variety of technologies and information resources in the process. Orders as we have observed them originate, are negotiated, and are carried out in a dynamically evolving group with fluctuating membership and shifting role responsibilities. Furthermore, orders by themselves represent only a partial picture of what is done for the patient.

CONCLUSION

We argue that information systems are more likely to be helpful if they accommodate and facilitate POE as a multidisciplinary collaboration effort and fit better into the larger system of patient care.

Perceptions of physician order entry: results of a cross-site qualitative study

OBJECTIVE

To identify perspectives of success factors for implementing computerized physician order entry (POE) in the inpatient setting.

DESIGN

Qualitative study by a multidisciplinary team using data from observation, focus groups, and both formal and informal interviews. Data were analyzed using a grounded approach to develop a taxonomy of patterns and themes from the transcripts and field notes.

RESULTS

A taxonomy of ten high level themes was developed, including 1) separating POE from other processes, 2) terms, concepts, and connotations, 3) context, 4) tradeoffs, 5) conflicts and contradictions, 6) collaboration and trust, 7) leaders and bridgers, 8) the organization of information, 9) the ongoing nature of implementation, and 10) temporal concerns.

CONCLUSION

The identified success factors indicate that POE implementation is an iterative and difficult process, but informants perceive it is worth the effort.

A cross-site qualitative study of physician order entry

OBJECTIVE

To describe the perceptions of diverse professionals involved in computerized physician order entry (POE) at sites where POE has been successfully implemented and to identify differences between teaching and nonteaching hospitals.

DESIGN

A multidisciplinary team used observation, focus groups, and interviews with clinical, administrative, and information technology staff to gather data at three sites. Field notes and transcripts were coded using an inductive approach to identify patterns and themes in the data.

MEASUREMENTS

Patterns and themes concerning perceptions of POE were identified.

RESULTS

Four high-level themes were identified: (1) organizational issues such as collaboration, pride, culture, power, politics, and control; (2) clinical and professional issues involving adaptation to local practices, preferences, and policies; (3) technical/implementation issues, including usability, time, training and support; and (4) issues related to the organization of information and knowledge, such as system rigidity and integration. Relevant differences between teaching and nonteaching hospitals include extent of collaboration, staff longevity, and organizational missions.

CONCLUSION

An organizational culture characterized by collaboration and trust and an ongoing process that includes active clinician engagement in adaptation of the technology were important elements in successful implementation of physician order entry at the institutions that we studied.

Using medication list--problem list mismatches as markers of potential error

The goal of this project was to specify and develop an algorithm that will check for drug and problem list mismatches in an electronic medical record (EMR). The algorithm is based on the premise that a patient's problem list and medication list should agree, and a mismatch may indicate medication error. Successful development of this algorithm could mean detection of some errors, such as medication orders entered into a wrong patient record, or drug therapy omissions, that are not otherwise detected via automated means. Additionally, mismatches may identify opportunities to improve problem list integrity. To assess the concept's feasibility, this study compared medications listed in a pharmacy information system with findings in an online nursing adult admission assessment, serving as a proxy for the problem list. Where drug and problem list mismatches were discovered, examination of the patient record confirmed the mismatch, and identified any potential causes. Evaluation of the algorithm in diabetes treatment indicates that it successfully detects both potential medication error and opportunities to improve problem list completeness. This algorithm, once fully developed and deployed, could prove a valuable way to improve the patient problem list, and could decrease the risk of medication error.

Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data

This paper explores the attributes of quality in recorded clinical encounter data, examines issues in measuring these attributes, and describes a method for measuring two attributes, completeness and correctness. The method is defined in the context of computer-based records and is demonstrated in a pilot study. Videotaped physician-patient encounters and an empiric process of determining a gold standard for content are used. The methodology was found to be feasible. Problems encountered during the pilot study can be remedied.

What's so special about medications: a pharmacist's observations from the POE study

Observations from a multi-site observational study of physician order entry (POE) confirm that implementing POE is problematic, and suggest that implementing medication order entry is particularly difficult. A pharmacist participating in the study group sought to answer the question: What makes medications different? Analysis of themes specific to medication POE in this study's large data set was undertaken using a grounded theory approach. Emerging themes in the data are explored and include: (1) order complexity and the consequences of error; (2) impacts on professional roles; (3) prescribing needs in different settings; and (4) technology impact on medication administration. Awareness of potential roadblocks and lessons learned from previous implementation attempts should help organizations considering medication POE to optimize their own strategies.

A taxonomy of generic clinical questions: classification study

OBJECTIVE

To develop a taxonomy of doctors' questions about patient care that could be used to help answer such questions.

DESIGN

Use of 295 questions asked by Oregon primary care doctors to modify previously developed taxonomy of 1101 clinical questions asked by Iowa family doctors.

SETTING

Primary care practices in Iowa and Oregon.

PARTICIPANTS

Random samples of 103 Iowa family doctors and 49 Oregon primary care doctors.

MAIN OUTCOME MEASURES

Consensus among seven investigators on a meaningful taxonomy of generic questions; interrater reliability among 11 individuals who used the taxonomy to classify a random sample of 100 questions: 50 from Iowa and 50 from Oregon.

RESULTS

The revised taxonomy, which comprised 64 generic question types, was used to classify 1396 clinical questions. The three commonest generic types were "What is the drug of choice for condition x?" (150 questions, 11%); "What is the cause of symptom x?" (115 questions, 8%); and "What test is indicated in situation x?" (112 questions, 8%). The mean interrater reliability among 11 coders was moderate (kappa=0.53, agreement 55%).

CONCLUSIONS

Clinical questions in primary care can be categorised into a limited number of generic types. A moderate degree of interrater reliability was achieved with the taxonomy developed in this study. The taxonomy may enhance our understanding of doctors' information needs and improve our ability to meet those needs.

Multiple perspectives on physician order entry

OBJECTIVE

Describe the complex interplay of perspectives of physicians, administrators, and information technology staff regarding computerized physician order entry (POE) in hospitals.

METHODS

Linstone's Multiple Perspectives Model provided a framework for organizing the results of a qualitative study done at four sites. Data from observation, focus groups, and formal and informal interviews were analyzed by four researchers using a grounded approach.

RESULTS

It is not a simple matter of physicians hating POE and others loving it. The issues involved are both complex and emotional. All groups see both positive and negative aspects of POE.

CONCLUSION

The Multiple Perspectives Model was useful for organizing a description to aid in understanding all points of view. It is imperative that those implementing POE understand all views and plan implementation strategies accordingly.

Perceptions of house officers who use physician order entry

OBJECTIVE

Describe the perceptions of housestaff physicians about their experience using computerized physician order entry (POE) in hospitals.

METHODS

Qualitative study using data from participant observation, focus groups, and both formal and informal interviews. Data were analyzed by three researchers using a grounded approach to identify patterns and themes in the texts.

RESULTS

Six themes were identified, including housestaff education, benefits of POE, problems with POE, feelings about POE, implementation strategies, and the future of POE.

CONCLUSION

House officers felt that POE assists patient care but may undermine education. They found that POE works best when tailored to fit local and individual workflow. Implementation strategies should include mechanisms for engaging housestaff in the decision process.

Physician order entry in U.S. hospitals

OBJECTIVE

Determine the percent of U.S. hospitals where computerized physician order entry (POE) is available and the extent of its use.

METHODS

A survey was sent to a systematic sample of 1,000 U.S. hospitals asking about availability of POE, whether usage is required, percent of physicians using it, and percent of orders entered by computer.

RESULTS

About 66% do not have POE available. Of the 32.1% that have it completely or partially available, 4.9% require its usage, over half report usage by under 10% of physicians, and over half report that fewer than 10% of orders are entered this way. Analysis of comments showed that many hospitals have POE available for use by non-physicians only, but that they hope to offer it to physicians after careful planning.

CONCLUSION

Most U.S. hospitals have not yet implemented POE. Complete availability throughout the hospital is rare, very few require its use, low percentages of physicians are actual users, and low percentages of orders are entered this way. On a national basis, computerized order entry by physicians is not yet widespread.

Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered

Primary care physicians have many questions about optimal care while they are seeing patients, but they pursue only about 30% of their questions. The authors designed a study to determine the factors that motivate physicians to pursue answers to some of their questions, while leaving the majority of their questions unanswered. They interviewed 49 non-academic primary care physicians during office hours to record clinical questions as they arose in the course of patient care. The physicians then recorded their perceptions of each question with respect to 12 factors expected to motivate information seeking. Two to five days after the interview, each physician was telephoned to determine which questions had been pursued. In a multiple logistic-regression model only two factors were significant predictors of pursuit of new information: the physician's belief that a definitive answer existed, and the urgency of the patient's problem. Other factors, including the difficulty of finding the answer, potential malpractice liability, potential help or harm to the patient, and self-perceived knowledge of the problem, were not significant in the model. Primary care physicians are significantly more likely to pursue answers to their clinical questions when they believe that definitive answers to those questions exist, and when they perceive the patient's problem to be urgent. Medical information systems must be shown to have direct and immediate benefits to solving the problems of patient care if they are to be more widely used by practitioners.

Can primary care physicians' questions be answered using the medical journal literature?

Medical librarians and informatics professionals believe the medical journal literature can be useful in clinical practice, but evidence suggests that practicing physicians do not share this belief. The authors designed a study to determine whether a random sample of "native" questions asked by primary care practitioners could be answered using the journal literature. Participants included forty-nine active, nonacademic primary care physicians providing ambulatory care in rural and nonrural Oregon, and seven medical librarians. The study was conducted in three stages: (1) office interviews with physicians to record clinical questions; (2) online searches to locate answers to selected questions; and (3) clinician feedback regarding the relevance and usefulness of the information retrieved. Of 295 questions recorded during forty-nine interviews, 60 questions were selected at random for searches. The average total time spent searching for and selecting articles for each question was forty-three minutes. The average cost per question searched was $27.37. Clinician feedback was received for 48 of 56 questions (four physicians could not be located, so their questions were not used in tabulating the results). For 28 questions (56%), clinicians judged the material relevant; for 22 questions (46%) the information provided a "clear answer" to their question. They expected the information would have had an impact on their patient in nineteen (40%) cases, and an impact on themselves or their practice in twenty-four (51%) cases. If the results can be generalized, and if the time and cost of performing searches can be reduced, increased use of the journal literature could significantly improve the extent to which primary care physicians' information needs are met.