The Impact of an Online Evidence System on Confidence in Decision Making in a Controlled Setting

Doctor! Did you Google my symptoms? A qualitative study of patient perceptions of doctors' point-of-care information seeking

OBJECTIVE

To explore patient perceptions regarding doctors' information seeking during consultations.

DESIGN AND SETTING

Qualitative interviews with participants from six general practice waiting rooms in South East Queensland, Australia. Participants were asked about their experiences and opinions, and to comment on short videos of simulated consultations in which a doctor sought information. The interviews were analysed through a process of iterative thematic analysis using the framework of Braun and Clarke.

PARTICIPANTS

The 16 participants were purposively sampled including 5 men and 11 women from a diverse range of educational and age groups.

RESULTS

How a doctor's need to look up information impacted patient impressions of competence and trust was an overarching theme. The four dominant themes include: the trust a patient has in the doctor before the consultation, whether the doctor is expected to know the answer to a question without searching, has the doctor added value to the consultation by searching and the consultation skills used in the process.

CONCLUSIONS

Patient trust is fundamental to positive perceptions of general practitioners' information seeking at the point-of-care. Communication is key to building this trust. Understanding the patient's agenda, listening, assessing thoroughly and being honest and transparent about the need to seek information all contribute to a positive experience.

Predicting alcohol dependence treatment outcomes: a prospective comparative study of clinical psychologists versus 'trained' machine learning models

BACKGROUND AND AIMS

Clinical staff are typically poor at predicting alcohol dependence treatment outcomes. Machine learning (ML) offers the potential to model complex clinical data more effectively. This study tested the predictive accuracy of ML algorithms demonstrated to be effective in predicting alcohol dependence outcomes, compared with clinical judgement and traditional linear regression.

DESIGN

Prospective study. ML models were trained on 1016 previously treated patients (training-set) who attended a hospital-based alcohol and drug clinic. ML models (n = 27), clinical psychologists (n = 10) and a 'traditional' logistic regression model (n = 1) predicted treatment outcome during the initial treatment session of an alcohol dependence programme.

SETTING

A 12-week cognitive behavioural therapy (CBT)-based abstinence programme for alcohol dependence in a hospital-based alcohol and drug clinic in Australia.

PARTICIPANTS

Prospective predictions were made for 220 new patients (test-set; 70.91% male, mean age = 35.78 years, standard deviation = 9.19). Sixty-nine (31.36%) patients successfully completed treatment.

MEASUREMENTS

Treatment success was the primary outcome variable. The cross-validated training-set accuracy of ML models was used to determine optimal parameters for selecting models for prospective prediction. Accuracy, sensitivity, specificity, area under the receiver operator curve (AUC), Brier score and calibration curves were calculated and compared across predictions.

FINDINGS

The mean aggregate accuracy of the ML models (63.06%) was higher than the mean accuracy of psychologist predictions (56.36%). The most accurate ML model achieved 70% accuracy, as did logistic regression. Both were more accurate than psychologists (P < 0.05) and had superior calibration. The high specificity for the selected ML (79%) and logistic regression (90%) meant they were significantly (P < 0.001) more effective than psychologists (50%) at correctly identifying patients whose treatment was unsuccessful. For ML and logistic regression, high specificity came at the expense of sensitivity (26 and 31%, respectively), resulting in poor prediction of successful patients.

CONCLUSIONS

Machine learning models and logistic regression appear to be more accurate than psychologists at predicting treatment outcomes in an abstinence programme for alcohol dependence, but sensitivity is low.

How searching under time pressure impacts clinical decision making

Objective:

Clinicians encounter many questions during patient encounters that they cannot answer. While search systems (e.g., PubMed) can help clinicians find answers, clinicians are typically busy and report that they often do not have sufficient time to use such systems. The objective of this study was to assess the impact of time pressure on clinical decisions made with the use of a medical literature search system.

Design:

In stage 1, 109 final-year medical students and practicing clinicians were presented with 16 clinical questions that they had to answer using their own knowledge. In stage 2, the participants were provided with a search system, similar to PubMed, to help them to answer the same 16 questions, and time pressure was simulated by limiting the participant's search time to 3, 6, or 9 minutes per question.

Results:

Under low time pressure, the correct answer rate significantly improved by 32% when the participants used the search system, whereas under high time pressure, this improvement was only 6%. Also, under high time pressure, participants reported significantly lower confidence in the answers, higher perception of task difficulty, and higher stress levels.

Conclusions:

For clinicians and health care organizations operating in increasingly time-pressured environments, literature search systems become less effective at supporting accurate clinical decisions. For medical search system developers, this study indicates that system designs that provide faster information retrieval and analysis, rather than traditional document search, may provide more effective alternatives.

Impact of a Search Engine on Clinical Decisions Under Time and System Effectiveness Constraints: Research Protocol

BACKGROUND

Many clinical questions arise during patient encounters that clinicians are unable to answer. An evidence-based medicine approach expects that clinicians will seek and apply the best available evidence to answer clinical questions. One commonly used source of such evidence is scientific literature, such as that available through MEDLINE and PubMed. Clinicians report that 2 key reasons why they do not use search systems to answer questions is that it takes too much time and that they do not expect to find a definitive answer. So, the question remains about how effectively scientific literature search systems support time-pressured clinicians in making better clinical decisions. The results of this study are important because they can help clinicians and health care organizations to better assess their needs with respect to clinical decision support (CDS) systems and evidence sources. The results and data captured will contribute a significant data collection to inform the design of future CDS systems to better meet the needs of time-pressured, practicing clinicians.

OBJECTIVE

The purpose of this study is to understand the impact of using a scientific medical literature search system on clinical decision making. Furthermore, to understand the impact of realistic time pressures on clinicians, we vary the search time available to find clinical answers. Finally, we assess the impact of improvements in search system effectiveness on the same clinical decisions.

METHODS

In this study, 96 practicing clinicians and final year medical students are presented with 16 clinical questions which they must answer without access to any external resource. The same questions are then represented to the clinicians; however, in this part of the study, the clinicians can use a scientific literature search engine to find evidence to support their answers. The time pressures of practicing clinicians are simulated by limiting answer time to one of 3, 6, or 9 min per question. The correct answer rate is reported both before and after search to assess the impact of the search system and the time constraint. In addition, 2 search systems that use the same user interface, but which vary widely in their search effectiveness, are employed so that the impact of changes in search system effectiveness on clinical decision making can also be assessed.

RESULTS

Recruiting began for the study in June 2018. As of the April 4, 2019, there were 69 participants enrolled. The study is expected to close by May 30, 2019, with results to be published in July.

CONCLUSIONS

All data collected in this study will be made available at the University of Queensland's UQ eSpace public data repository.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/12803.

Machine learning vs addiction therapists: A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication

BACKGROUND AND OBJECTIVES

Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only.

METHODS

Machine learning models (n = 28) were constructed ('trained') using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined.

RESULTS

The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74%) was significantly more accurate than the four least accurate clinical staff (51%-40%). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1%) and the 28 best models (58.57%). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales.

CONCLUSIONS

The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.

Care episode retrieval: distributional semantic models for information retrieval in the clinical domain

Patients' health related information is stored in electronic health records (EHRs) by health service providers. These records include sequential documentation of care episodes in the form of clinical notes. EHRs are used throughout the health care sector by professionals, administrators and patients, primarily for clinical purposes, but also for secondary purposes such as decision support and research. The vast amounts of information in EHR systems complicate information management and increase the risk of information overload. Therefore, clinicians and researchers need new tools to manage the information stored in the EHRs. A common use case is, given a--possibly unfinished--care episode, to retrieve the most similar care episodes among the records. This paper presents several methods for information retrieval, focusing on care episode retrieval, based on textual similarity, where similarity is measured through domain-specific modelling of the distributional semantics of words. Models include variants of random indexing and the semantic neural network model word2vec. Two novel methods are introduced that utilize the ICD-10 codes attached to care episodes to better induce domain-specificity in the semantic model. We report on experimental evaluation of care episode retrieval that circumvents the lack of human judgements regarding episode relevance. Results suggest that several of the methods proposed outperform a state-of-the art search engine (Lucene) on the retrieval task.

Improving diagnostic accuracy using EHR in emergency departments: A simulation-based study

It is widely believed that Electronic Health Records (EHR) improve medical decision-making by enabling medical staff to access medical information stored in the system. It remains unclear, however, whether EHR indeed fulfills this claim under the severe time constraints of Emergency Departments (EDs). We assessed whether accessing EHR in an ED actually improves decision-making by clinicians. A simulated ED environment was created at the Israel Center for Medical Simulation (MSR). Four different actors were trained to simulate four specific complaints and behavior and 'consulted' 26 volunteer ED physicians. Each physician treated half of the cases (randomly) with access to EHR, and their medical decisions were compared to those where the physicians had no access to EHR. Comparison of diagnostic accuracy with and without access showed that accessing the EHR led to an increase in the quality of the clinical decisions. Physicians accessing EHR were more highly informed and thus made more accurate decisions. The percentage of correct diagnoses was higher and these physicians were more confident in their diagnoses and made their decisions faster.

Evaluating a federated medical search engine: tailoring the methodology and reporting the evaluation outcomes

BACKGROUND

Federated medical search engines are health information systems that provide a single access point to different types of information. Their efficiency as clinical decision support tools has been demonstrated through numerous evaluations. Despite their rigor, very few of these studies report holistic evaluations of medical search engines and even fewer base their evaluations on existing evaluation frameworks.

OBJECTIVES

To evaluate a federated medical search engine, MedSocket, for its potential net benefits in an established clinical setting.

METHODS

This study applied the Human, Organization, and Technology (HOT-fit) evaluation framework in order to evaluate MedSocket. The hierarchical structure of the HOT-factors allowed for identification of a combination of efficiency metrics. Human fit was evaluated through user satisfaction and patterns of system use; technology fit was evaluated through the measurements of time-on-task and the accuracy of the found answers; and organization fit was evaluated from the perspective of system fit to the existing organizational structure.

RESULTS

Evaluations produced mixed results and suggested several opportunities for system improvement. On average, participants were satisfied with MedSocket searches and confident in the accuracy of retrieved answers. However, MedSocket did not meet participants' expectations in terms of download speed, access to information, and relevance of the search results. These mixed results made it necessary to conclude that in the case of MedSocket, technology fit had a significant influence on the human and organization fit. Hence, improving technological capabilities of the system is critical before its net benefits can become noticeable.

CONCLUSIONS

The HOT-fit evaluation framework was instrumental in tailoring the methodology for conducting a comprehensive evaluation of the search engine. Such multidimensional evaluation of the search engine resulted in recommendations for system improvement.

Using electronic health record systems to optimize admission decisions: the Creatinine case study

Many medical organizations have implemented electronic health record (EHR) and health information exchange (HIE) networks to improve medical decision-making. This study evaluated the contribution of EHR and HIE networks to physicians by investigating whether health information technology can lead to more efficient admission decisions by reducing redundant admissions in the stressful environment of emergency. Log-files were retrieved from an integrative and interoperable EHR that serves seven main Israeli hospitals. The analysis was restricted to a group of patients seen in the emergency departments who were administered a Creatinine test. The assessment of the contribution of EHR to admission decisions used various statistical analyses and track log-file analysis. We showed that using the EHR contributes to more efficient admission decisions and reduces the number of avoidable admissions. In particular, there was a reduction in readmissions when patient history was viewed. Using EHR can help respond to the international problem of avoidable hospital readmissions.

How do medical doctors use a web-based oncology protocol system? A comparison of Australian doctors at different levels of medical training using logfile analysis and an online survey

BACKGROUND

Electronic decision support is commonplace in medical practice. However, its adoption at the point-of-care is dependent on a range of organisational, patient and clinician-related factors. In particular, level of clinical experience is an important driver of electronic decision support uptake. Our objective was to examine the way in which Australian doctors at different stages of medical training use a web-based oncology system (http://www.eviq.org.au).

METHODS

We used logfiles to examine the characteristics of eviQ registrants (2009-2012) and patterns of eviQ use in 2012, according to level of medical training. We also used a web-based survey to evaluate the way doctors at different levels of medical training use the online system and to elicit perceptions of the system's utility in oncology care.

RESULTS

Our study cohort comprised 2,549 eviQ registrants who were hospital-based medical doctors across all levels of training. 65% of the cohort used eviQ in 2012, with 25% of interns/residents, 61% of advanced oncology trainees and 47% of speciality-qualified oncologists accessing eviQ in the last 3 months of 2012. The cohort accounted for 445,492 webhits in 2012. On average, advanced trainees used eviQ up to five-times more than other doctors (42.6 webhits/month compared to 22.8 for specialty-qualified doctors and 7.4 webhits/month for interns/residents). Of the 52 survey respondents, 89% accessed eviQ's chemotherapy protocols on a daily or weekly basis in the month prior to the survey. 79% of respondents used eviQ at least weekly to initiate therapy and to support monitoring (29%), altering (35%) or ceasing therapy (19%). Consistent with the logfile analysis, advanced oncology trainees report more frequent eviQ use than doctors at other stages of medical training.

CONCLUSIONS

The majority of the Australian oncology workforce are registered on eviQ. The frequency of use directly mirrors the clinical role of doctors and attitudes about the utility of eviQ in decision-making. Evaluations of this kind generate important data for system developers and medical educators to drive improvements in electronic decision support to better meet the needs of clinicians. This end-user focus will optimise the uptake of systems which will translate into improvements in processes of care and patient outcomes.

The role of ICT in supporting disruptive innovation: a multi-site qualitative study of nurse practitioners in emergency departments

BACKGROUND

The disruptive potential of the Nurse Practitioner (NP) is evident in their ability to offer services traditionally provided by primary care practitioners and their provision of a health promotion model of care in response to changing health trends. No study has qualitatively investigated the role of the Emergency NP in Australia, nor the impact of Information and Communication Technology (ICT) on this disruptive workforce innovation. This study aimed to investigate ways in which Nurse Practitioners (NP) have incorporated the use of ICT as a mechanism to support their new clinical role within Emergency Departments.

METHODS

A cross-sectional qualitative study was undertaken in the Emergency Departments (EDs) of two large Australian metropolitan public teaching hospitals. Semi-structured, in-depth interviews were conducted with five nurse practitioners, four senior physicians and five senior nurses. Transcribed interviews were analysed using a grounded theory approach to develop themes in relation to the conceptualisation of the ED nurse practitioner role and the influences of ICT upon the role. Member checking of results was achieved by revisiting the sites to clarify findings with participants and further explore emergent themes.

RESULTS

The role of the ENP was distinguished from those of Emergency nurses and physicians by two elements: advanced practice and holistic care, respectively. ICT supported the advanced practice dimension of the NP role in two ways: availability and completeness of electronic patient information enhanced timeliness and quality of diagnostic and therapeutic decision-making, expediting patient access to appropriate care. The ubiquity of patient data sourced from a central database supported and improved quality of communication between health professionals within and across sites, with wider diffusion of the Electronic Medical Record holding the potential to further facilitate team-based, holistic care.

CONCLUSIONS

ICT is a facilitator through which the disruptive impact of NPs can be extended. However, integration of ICT into work practices without detracting from provider-patient interaction is crucial to ensure utilisation of such interventions and realisation of potential benefits.

How online crowds influence the way individual consumers answer health questions: an online prospective study

OBJECTIVE

To investigate whether strength of social feedback, i.e. other people who concur (or do not concur) with one's own answer to a question, influences the way one answers health questions.

METHODS

Online prospective study. Two hundred and twenty-seven undergraduate students were recruited to use an online search engine to answer six health questions. Subjects recorded their pre- and post-search answers to each question and their level of confidence in these answers. After answering each question post-search, subjects were presented with a summary of post-search answers provided by previous subjects and were asked to answer the question again.

RESULTS

There was a statistically significant relationship between the absolute number of others with a different answer (the crowd's opinion volume) and the likelihood of an individual changing an answer (P<0.001). For most questions, no subjects changed their answer until the first 10-35 subjects completed the study. Subjects' likelihood of changing answer increased as the percentage of others with a different answer (the crowd's opinion density) increased (P=0.047). Overall, 98.3% of subjects did not change their answer when it concurred with the majority (i.e. >50%) of subjects, and that 25.7% of subjects changed their answer to the majority response when it did not concur with the majority. When subjects had a post-search answer that did not concur with the majority, they were 24% more likely to change answer than those with answers that concurred (P<0.001).

CONCLUSION

This study provides empirical evidence that crowd influence, in the form of online social feedback, affects the way consumers answer health questions.

Will information and communication technology disrupt the health system and deliver on its promise?

Investment in information and communication technology (ICT) in the health sector can bring important benefits. To date, the focus has been on automating clinical work practices such as ordering tests and prescriptions, which significantly improves efficiency and safety. Uptake of ICT has been slow and the results less favourable than anticipated for various reasons, including poor integration of systems into complex clinical work processes, limited training, and the intermittent nature of ICT funding. As a result, many health care organisations have been operating hybrid paper and computer systems that introduce new patient risks, staff frustration, and outcomes below expectation. The focus must shift from automation of clinical work to innovation; from evolutionary application of ICT to revolutionary uses. Health professionals must embrace ICT as a "disruptive technology" that will produce significant changes in their roles and responsibilities and lead to real health reform with new, innovative models of health care delivery. As other industries have shown, substitution and role changes are areas in which ICT can lead to the greatest gains.

Encouraging physicians to screen for and intervene in substance use disorders: obstacles and strategies for change

The scientific literature was reviewed to identify obstacles and effective ways to improve primary care physician screening, interventions, and management of patient substance use disorders (SUDs). Major obstacles identified are physician lack of skills and self-efficacy in patient counseling, inadequate training at all levels of medical education, and lack of reimbursement and other health care systems support for services to patients. Physician abuse of drugs does not appear to be a major obstacle. Physician attitudes about patients with SUDS and the effectiveness of treatment services need to be addressed. Research points to the use of a multifaceted change strategy. Key components include practice-based training emphasizing screening and counseling skills throughout medical education, clinical systems to ensure regular SUD services (screening, intervention, and referral) to patients, and reimbursement and coverage systems to support physician interventions and patient services.

Can cognitive biases during consumer health information searches be reduced to improve decision making?

OBJECTIVE

To test whether the anchoring and order cognitive biases experienced during search by consumers using information retrieval systems can be corrected to improve the accuracy of, and confidence in, answers to health-related questions.

DESIGN

A prospective study was conducted on 227 undergraduate students who used an online search engine developed by the authors to find health information and then answer six randomly assigned consumer health questions. The search engine was fitted with a baseline user interface and two modified interfaces specifically designed to debias anchoring or order effect. Each subject used all three user interfaces, answering two questions with each.

MEASUREMENTS

Frequencies of correct answers pre- and post- search and confidence in answers were collected. Time taken to search and then answer a question, the number of searches conducted and the number of links accessed in a search session were also recorded. User preferences for each interface were measured. Chi-square analyses tested for the presence of biases with each user interface. The Kolmogorov-Smirnov test checked for equality of distribution of the evidence analyzed for each user interface. The test for difference between proportions and the Wilcoxon signed ranks test were used when comparing interfaces.

RESULTS

Anchoring and order effects were present amongst subjects using the baseline search interface (anchoring: p < 0.001; order: p = 0.026). With use of the order debiasing interface, the initial order effect was no longer present (p = 0.34) but there was no significant improvement in decision accuracy (p = 0.23). While the anchoring effect persisted when using the anchor debiasing interface (p < 0.001), its use was associated with a 10.3% increase in subjects who had answered incorrectly pre-search, answering correctly post-search (p = 0.10). Subjects using either debiasing user interface conducted fewer searches and accessed more documents compared to baseline (p < 0.001). In addition, the majority of subjects preferred using a debiasing interface over baseline.

CONCLUSION

This study provides evidence that (i) debiasing strategies can be integrated into the user interface of a search engine; (ii) information interpretation behaviors can be to some extent debiased; and that (iii) attempts to debias information searching by consumers can influence their ability to answer health-related questions accurately, their confidence in these answers, as well as the strategies used to conduct searches and retrieve information.

Effectiveness of topic-specific infobuttons: a randomized controlled trial

OBJECTIVE

Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links").

DESIGN

Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links.

MEASUREMENTS

Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making.

RESULTS

The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias.

CONCLUSION

The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

Clinical decision velocity is increased when meta-search filters enhance an evidence retrieval system

OBJECTIVE

To test whether the use of an evidence retrieval system that uses clinically targeted meta-search filters can enhance the rate at which clinicians make correct decisions, reduce the effort involved in locating evidence, and provide an intuitive match between clinical tasks and search filters.

DESIGN

A laboratory experiment under controlled conditions asked 75 clinicians to answer eight randomly sequenced clinical questions, using one of two randomly assigned search engines. The first search engine Quick Clinical (QC) was equipped with meta-search filters (the combined use of meta-search and search filters) designed to answer typical clinical questions e.g., treatment, diagnosis, and the second 'library model' system (LM) offered free access to an identical evidence set with no filter support.

MEASUREMENTS

Changes in clinical decision making were measured by the proportion of correct post-search answers provided to questions, the time taken to answer questions, and the number of searches and links to documents followed in a search session. The intuitive match between meta-search filters and clinical tasks was measured by the proportion and distribution of filters selected for individual clinical questions.

RESULTS

Clinicians in the two groups performed equally well pre-search. Post search answers improved overall by 21%, with 52.2% of answers correct with QC and 54.7% with LM (chi(2) = 0.33, df = 1, p > 0.05). Users of QC obtained a significantly greater percentage of their correct answers within the first two minutes of searching compared to LM users (QC 58.2%; LM 32.9%; chi(2) = 19.203, df = 1, p < 0.001). There was a statistical difference for QC and LM survival curves, which plotted overall time to answer questions, irrespective of answer (Wilcoxon, p = 0.019) and for the average time to provide a correct answer (Wilcoxon, p = 0.006). The QC system users conducted significantly fewer searches per scenario (m = 3.0 SD = 1.15 versus m = 5.5 SD1.97, t = 6.63, df = 72, p = 0.0001). Clinicians using the QC system followed fewer document links than did those who used LM (respectively 3.9 links SD = 1.20 versus 4.7 links SD = 1.79, t = 2.13, df = 72, p = 0.0368). In 6 of the 8 questions, two meta-search filters accounted for 89% or more of clinicians' first choice, suggesting the choice of filter intuitively matched the clinical decision task at hand.

CONCLUSIONS

Meta-search filters result in clinicians arriving at answers more quickly than unconstrained searches across information sources, and appear to increase the rate with which correct decisions are made. In time restricted clinical settings meta-search filters may thus improve overall decision accuracy, as fewer searches that could otherwise lead to a correct answer are abandoned. Meta-search filters appear to be intuitive to use, suggesting that the simplicity of the user model would fit very well into clinical settings.

Impact of web searching and social feedback on consumer decision making: a prospective online experiment

BACKGROUND

The World Wide Web has increasingly become an important source of information in health care consumer decision making. However, little is known about whether searching online resources actually improves consumers' understanding of health issues.

OBJECTIVES

The aim was to study whether searching on the World Wide Web improves consumers' accuracy in answering health questions and whether consumers' understanding of health issues is subject to further change under social feedback.

METHODS

This was a pre/post prospective online study. A convenience sample of 227 undergraduate students was recruited from the population of the University of New South Wales. Subjects used a search engine that retrieved online documents from PubMed, MedlinePlus, and HealthInsite and answered a set of six questions (before and after use of the search engine) designed for health care consumers. They were then presented with feedback consisting of a summary of the post-search answers provided by previous subjects for the same questions and were asked to answer the questions again.

RESULTS

There was an improvement in the percentage of correct answers after searching (pre-search 61.2% vs post-search 82.0%, P <.001) and after feedback with other subjects' answers (pre-feedback 82.0% vs post-feedback 85.3%, P =.051). The proportion of subjects with highly confident correct answers (ie, confident or very confident) and the proportion with highly confident incorrect answers significantly increased after searching (correct pre-search 61.6% vs correct post-search 95.5%, P <.001; incorrect pre-search 55.3% vs incorrect post-search 82.0%, P <.001). Subjects who were not as confident in their post-search answers were 28.5% more likely than those who were confident or very confident to change their answer after feedback with other subjects' post-search answers (chi(2) (1)= 66.65, P <.001).

CONCLUSIONS

Searching across quality health information sources on the Web can improve consumers' accuracy in answering health questions. However, a consumer's confidence in an answer is not a good indicator of the answer being correct. Consumers who are not confident in their answers after searching are more likely to be influenced to change their views when provided with feedback from other consumers.

What factors are associated with the integration of evidence retrieval technology into routine general practice settings?

BACKGROUND

Information retrieval systems have the potential to improve patient care but little is known about the variables which influence clinicians' uptake and use of systems in routine work.

AIM

To determine which factors influenced use of an online evidence retrieval system.

DESIGN OF STUDY

Computer logs and pre- and post-system survey analysis of a 4-week clinical trial of the Quick Clinical online evidence system involving 227 general practitioners across Australia.

RESULTS

Online evidence use was not linked to general practice training or clinical experience but female clinicians conducted more searches than their male counterparts (mean use=14.38 searches, S.D.=11.68 versus mean use=8.50 searches, S.D.=9.99; t=2.67, d.f.=157, P=0.008). Practice characteristics such as hours worked, type and geographic location of clinic were not associated with search activity. Information seeking was also not related to participants' perceived information needs, computer skills, training nor Internet connection speed. Clinicians who reported direct improvements in patient care as a result of system use had significantly higher rates of system use than other users (mean use=12.55 searches, S.D.=13.18 versus mean use=8.15 searches, S.D.=9.18; t=2.322, d.f.=154 P=0.022). Comparison of participants' views pre- and post- the trial, showed that post-trial clinicians expressed more positive views about searching for information during a consultation (chi(2)=27.40, d.f.=4, P< or =0.001) and a significantly greater number reported seeking information between consultations as a result of having access to an online evidence system in their consulting rooms (chi(2)=9.818, d.f.=2, P=0.010).

CONCLUSION

Clinicians' use of an online evidence system was directly related to their reported experiences of improvements in patient care. Post-trial clinicians positively changed their views about having time to search for information and pursued more questions during clinic hours.

Effectiveness of clinician-selected electronic information resources for answering primary care physicians' information needs

OBJECTIVE

To determine if clinician-selected electronic information resources improve primary care physicians' abilities to answer simulated clinical questions.

DESIGN

Observational study using hour-long interviews in physician offices and think-aloud protocols.

PARTICIPANTS

answered 23 multiple-choice questions and chose 2 to obtain further information using their own information resources. We established which resources physicians chose, processes used, and results obtained when looking for information to support their answers.

MEASUREMENTS

Correctness of answers before and after searching, resources used, and searching techniques.

RESULTS

23 physicians sought answers to 46 questions using their own information resources. They spent a mean of 13.0 (SD 5.5) minutes searching for information for the two questions using an average of 1.8 resources per question and a wide variety of searching techniques. On average 43.5% of the answers to the original 23 questions were correct. For the questions that were searched, 18 (39.1%) of the 46 answers were correct before searching. After searching, the number of correct answers was 19 (42.1%). This difference of 1 correct answer was attributed to 6 questions (13.0%) going from an incorrect to correct answer and 5 (10.9%) questions going from a correct to incorrect answer. We found differences in the ability of various resources to provide correct answers.

CONCLUSION

For the primary care physicians studied, electronic information resources of choice did not always provide support for finding correct answers to simulated clinical questions and in some instances, individual resources may have contributed to an initially correct answer becoming incorrect.

Ambulance officers' use of online clinical evidence

Background

Hospital-based clinicians have been shown to use and attain benefits from online evidence systems. To our knowledge there have been no studies investigating whether and how ambulance officers use online evidence systems if provided. We surveyed ambulance officers to examine their knowledge and use of the Clinical Information Access Program (CIAP), an online evidence system providing 24-hour access to information to support evidence-based practice.

Methods

A questionnaire was completed by 278 ambulance officers in New South Wales, Australia. Comparisons were made between those who used CIAP and officers who had heard of, but not used CIAP.

Results

Half the sample (48.6%) knew of, and 28.8% had used CIAP. Users were more likely to have heard of CIAP from a CIAP representative/presentation, non-users from written information. Compared to ambulance officers who had heard of but had not used CIAP, users were more likely to report better computer skills and that their supervisors regarded use of CIAP as a legitimate part of ambulance officers' clinical role. The main reasons for non-use were lack of access(49.0%) and training(31.4%). Of users, 51.3% rated their skills at finding information as good/very good, 67.5% found the information sought all/most of the time, 87.3% believed CIAP had the potential to improve patient care and 28.2% had directly experienced this. Most access to CIAP occurred at home. The databases frequently accessed were MIMS (A medicines information database) (73.8%) and MEDLINE(67.5%). The major journals accessed were Journal of Emergency Nursing(37.5%), American Journal of Medicine(30.0%) and JAMA(27.5%).

Conclusion

Over half of ambulance officers had not heard of CIAP. The proportion who knew about and used CIAP was also low. Reasons for this appear to be a work culture not convinced of CIAP's relevance to pre-hospital patient care and lack of access to CIAP at work. Ambulance officers who used CIAP accessed it primarily from home and valued it highly. Lack of access to CIAP at central work locations deprives ambulance officers of many of the benefits of an online evidence system.

Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes

BACKGROUND

Online evidence retrieval systems are a potential tool in supporting evidence-based practice. Effective and tested techniques for assessing the impact of these systems on care delivery and patient outcomes are limited.

PURPOSE AND METHODS

In this study we applied the critical incident (CI) and journey mapping (JM) techniques to assess the integration of an online evidence system into everyday clinical practice and its impact on decision making and patient care. To elicit incidents semi-structured interviews were conducted with 29 clinicians (13 hospital physician specialists, 16 clinical nurse consultants (CNCs)) who were experienced users of the online evidence system. Clinicians were also asked questions about how they had first used the system and how their use and experiences had changed over time. These narrative accounts were then mapped and scored using the journey mapping technique.

RESULTS

Clinicians generated 85 critical incidents. Three categories of impact were identified: impact on clinical practice, impact on individual clinicians and impact on colleagues through the dissemination of information gained from the online evidence system. One quarter of these included specific examples of system use leading to improvements in patient care. Clinicians obtained an average journey mapping score of 22 out of a possible score of 36, demonstrating a good level of system integration. Average scores of doctors and CNCs were similar. However individuals with the same scores often had very different journeys in system integration.

CONCLUSIONS

The CI technique provided clear examples of the way in which system use had influenced practice and care delivery. The JM technique was found to be a useful method for providing a quantification of the different ways and extent to which, clinicians had integrated system use into practice, and insights into how system use can influence organisational culture. The development of the journey mapping stages provides a structure by which the program logic of a clinical information system and its desired outcomes can be made explicit and be based upon users' experiences in everyday practice. Further work is required using this technique to assess its value as an evaluation method.

Architecture for knowledge-based and federated search of online clinical evidence

Background

It is increasingly difficult for clinicians to keep up-to-date with the rapidly growing biomedical literature. Online evidence retrieval methods are now seen as a core tool to support evidence-based health practice. However, standard search engine technology is not designed to manage the many different types of evidence sources that are available or to handle the very different information needs of various clinical groups, who often work in widely different settings.

Objectives

The objectives of this paper are (1) to describe the design considerations and system architecture of a wrapper-mediator approach to federate search system design, including the use of knowledge-based, meta-search filters, and (2) to analyze the implications of system design choices on performance measurements.

Methods

A trial was performed to evaluate the technical performance of a federated evidence retrieval system, which provided access to eight distinct online resources, including e-journals, PubMed, and electronic guidelines. The Quick Clinical system architecture utilized a universal query language to reformulate queries internally and utilized meta-search filters to optimize search strategies across resources. We recruited 227 family physicians from across Australia who used the system to retrieve evidence in a routine clinical setting over a 4-week period. The total search time for a query was recorded, along with the duration of individual queries sent to different online resources.

Results

Clinicians performed 1662 searches over the trial. The average search duration was 4.9 ± 3.2 s (N = 1662 searches). Mean search duration to the individual sources was between 0.05 s and 4.55 s. Average system time (ie, system overhead) was 0.12 s.

Conclusions

The relatively small system overhead compared to the average time it takes to perform a search for an individual source shows that the system achieves a good trade-off between performance and reliability. Furthermore, despite the additional effort required to incorporate the capabilities of each individual source (to improve the quality of search results), system maintenance requires only a small additional overhead.

Use of point-of-care online clinical evidence by junior and senior doctors in New South Wales public hospitals

BACKGROUND

The Clinical Information Access Program (CIAP), an online evidence retrieval system, provides NSW health professionals in public hospitals with 24 h access to information supporting evidence-based practice.

AIM

To assess the frequency and type of CIAP usage by senior and junior medical staff and doctors' attitudes to CIAP.

METHODS

A convenience sample of 25% of doctors from 65 randomly selected public hospitals completed a survey. Junior (n = 392) and senior (n = 684) doctors' responses were compared using chi2 analyses and t-tests.

RESULTS

Most doctors had heard of CIAP (71.8%) and 60.6% had used it. More junior (72.4%) than senior (53.8%) doctors had used CIAP and junior doctors found it easier to use. Of the users 93.5% believed CIAP had the potential to improve patient care; 55.2% had directly experienced this. Most usage (61.5%) occurred at point-of-care; 74% of users found all/most of the information they sought and 71.6% found the search time to be 'good/excellent'. Users had increased their usage in the past year and predicted increased future usage. The most popular databases were Medline and MIMS. Age, access to other evidence, and lack of training, time and computer skills were associated with non-usage. Junior and senior users differed in 4 of 15 reasons for using CIAP.

CONCLUSIONS

CIAP is used and valued by the majority of doctors. Patterns of usage, online experiences and the attitudes toward CIAP of senior doctors who use CIAP are relatively similar to those of junior doctors.

General practitioners' use of online evidence during consultations

BACKGROUND

Clinicians have many unanswered questions during clinical encounters which may impact on the quality and outcomes of decisions made. Provision of online evidence at the point of care is one strategy that provides clinicians with easy access to up-to-date evidence in clinical settings to support evidence-based decision-making.

AIM

To determine if and when general practitioners use an online evidence system in routine clinical practice, the type of questions for which clinicians seek evidence and the extent to which the system provides clinically useful answers.

DESIGN OF STUDY

A prospective cohort study which involved a 4-week clinical trial of Quick Clinical, an online evidence system specifically designed around the needs of general practitioners.

SETTING

Two hundred and twenty-seven clinicians who had a computer with Internet access in their consulting rooms.

METHODS

Computer logs and survey analysis.

RESULTS

One hundred and ninety-three general practitioners used the online evidence system to conduct on average 8.7 searches/month. The majority of these (81%) were conducted from consulting rooms and carried out between 9a.m. and 7p.m. (83%). The most frequent searches conducted related to diagnosis (40%) and treatment (35%). 83% of clinicians believed that Quick Clinical (QC) had the potential to improve patient care, and one in four users reported direct experience of improvements in care. In 73% of queries with clinician feedback participants reported that they were able to find clinically useful information during their routine work.

CONCLUSION

General practitioners will use an online evidence retrieval system in routine practice, and report that its use improves the quality of patient care.

Do online information retrieval systems help experienced clinicians answer clinical questions?

OBJECTIVE

To assess the impact of clinicians' use of an online information retrieval system on their performance in answering clinical questions.

DESIGN

Pre-/post-intervention experimental design.

MEASUREMENTS

In a computer laboratory, 75 clinicians (26 hospital-based doctors, 18 family practitioners, and 31 clinical nurse consultants) provided 600 answers to eight clinical scenarios before and after the use of an online information retrieval system. We examined the proportion of correct answers pre- and post-intervention, direction of change in answers, and differences between professional groups.

RESULTS

System use resulted in a 21% improvement in clinicians' answers, from 29% (95% confidence interval [CI] 25.4-32.6) correct pre- to 50% (95% CI 46.0-54.0) post-system use. In 33% (95% CI 29.1-36.9) answers were changed from incorrect to correct. In 21% (95% CI 17.1-23.9) correct pre-test answers were supported by evidence found using the system, and in 7% (95% CI 4.9-9.1) correct pre-test answers were changed incorrectly. For 40% (35.4-43.6) of scenarios, incorrect pre-test answers were not rectified following system use. Despite significant differences in professional groups' pre-test scores [family practitioners: 41% (95% CI 33.0-49.0), hospital doctors: 35% (95% CI 28.5-41.2), and clinical nurse consultants: 17% (95% CI 12.3-21.7; chi(2) = 29.0, df = 2, p < 0.01)], there was no difference in post-test scores. (chi(2) = 2.6, df = 2, p = 0.73).

CONCLUSIONS

The use of an online information retrieval system was associated with a significant improvement in the quality of answers provided by clinicians to typical clinical problems. In a small proportion of cases, use of the system produced errors. While there was variation in the performance of clinical groups when answering questions unaided, performance did not differ significantly following system use. Online information retrieval systems can be an effective tool in improving the accuracy of clinicians' answers to clinical questions.