A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics

One Health: Insights from Organizational & Social, Technology Assessment and Human Factors Perspectives

OBJECTIVES

To offer diverse but complementary perspectives on how biomedical and health informatics can be informed by and help to achieve the vision of One Health.

METHODS

Overview of key considerations and critical discussion of common themes, barriers and opportunities, based on collaborative review by International Medical Informatics Association (IMIA) working group members active in related fields.

RESULTS

Health and care systems are complex sociotechnical systems that need explicit design and implementation strategies to align with the goals of One Health. The evidence-based health informatics paradigm and associated frameworks for evaluation of digital health technologies need to broaden their scope to take full account of the One Health approach. Informatics has specific contributions to make to One Health, for example by improved user experience reducing energy consumption and effective app design enhancing medication adherence.

CONCLUSIONS

One Health is inherently intertwined with ergonomic, sociotechnical and evaluation perspectives in biomedical and health informatics. Health is a planetary issue that requires interdisciplinary collaborative action. The theories and principles of biomedical and health informatics offer many opportunities to transform digital health technology to better serve the One Health agenda.

Comparing Decentralized Learning Methods for Health Data Models to Nondecentralized Alternatives: Protocol for a Systematic Review

BACKGROUND

Considering the soaring health-related costs directed toward a growing, aging, and comorbid population, the health sector needs effective data-driven interventions while managing rising care costs. While health interventions using data mining have become more robust and adopted, they often demand high-quality big data. However, growing privacy concerns have hindered large-scale data sharing. In parallel, recently introduced legal instruments require complex implementations, especially when it comes to biomedical data. New privacy-preserving technologies, such as decentralized learning, make it possible to create health models without mobilizing data sets by using distributed computation principles. Several multinational partnerships, including a recent agreement between the United States and the European Union, are adopting these techniques for next-generation data science. While these approaches are promising, there is no clear and robust evidence synthesis of health care applications.

OBJECTIVE

The main aim is to compare the performance among health data models (eg, automated diagnosis and mortality prediction) developed using decentralized learning approaches (eg, federated and blockchain) to those using centralized or local methods. Secondary aims are comparing the privacy compromise and resource use among model architectures.

METHODS

We will conduct a systematic review using the first-ever registered research protocol for this topic following a robust search methodology, including several biomedical and computational databases. This work will compare health data models differing in development architecture, grouping them according to their clinical applications. For reporting purposes, a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 flow diagram will be presented. CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies)-based forms will be used for data extraction and to assess the risk of bias, alongside PROBAST (Prediction Model Risk of Bias Assessment Tool). All effect measures in the original studies will be reported.

RESULTS

The queries and data extractions are expected to start on February 28, 2023, and end by July 31, 2023. The research protocol was registered with PROSPERO, under the number 393126, on February 3, 2023. With this protocol, we detail how we will conduct the systematic review. With that study, we aim to summarize the progress and findings from state-of-the-art decentralized learning models in health care in comparison to their local and centralized counterparts. Results are expected to clarify the consensuses and heterogeneities reported and help guide the research and development of new robust and sustainable applications to address the health data privacy problem, with applicability in real-world settings.

CONCLUSIONS

We expect to clearly present the status quo of these privacy-preserving technologies in health care. With this robust synthesis of the currently available scientific evidence, the review will inform health technology assessment and evidence-based decisions, from health professionals, data scientists, and policy makers alike. Importantly, it should also guide the development and application of new tools in service of patients' privacy and future research.

TRIAL REGISTRATION

PROSPERO 393126; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=393126.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/45823.

Connected Mental Health: Systematic Mapping Study

BACKGROUND

Although mental health issues constitute an increasing global burden affecting a large number of people, the mental health care industry is still facing several care delivery barriers such as stigma, education, and cost. Connected mental health (CMH), which refers to the use of information and communication technologies in mental health care, can assist in overcoming these barriers.

OBJECTIVE

The aim of this systematic mapping study is to provide an overview and a structured understanding of CMH literature available in the Scopus database.

METHODS

A total of 289 selected publications were analyzed based on 8 classification criteria: publication year, publication source, research type, contribution type, empirical type, mental health issues, targeted cohort groups, and countries where the empirically evaluated studies were conducted.

RESULTS

The results showed that there was an increasing interest in CMH publications; journals were the main publication channels of the selected papers; exploratory research was the dominant research type; advantages and challenges of the use of technology for mental health care were the most investigated subjects; most of the selected studies had not been evaluated empirically; depression and anxiety were the most addressed mental disorders; young people were the most targeted cohort groups in the selected publications; and Australia, followed by the United States, was the country where most empirically evaluated studies were conducted.

CONCLUSIONS

CMH is a promising research field to present novel approaches to assist in the management, treatment, and diagnosis of mental health issues that can help overcome existing mental health care delivery barriers. Future research should be shifted toward providing evidence-based studies to examine the effectiveness of CMH solutions and identify related issues.

Publication and related bias in quantitative health services and delivery research: a multimethod study

Background

Bias in the publication and reporting of research findings (referred to as publication and related bias here) poses a major threat in evidence synthesis and evidence-based decision-making. Although this bias has been well documented in clinical research, little is known about its occurrence and magnitude in health services and delivery research.

Objectives

To obtain empirical evidence on publication and related bias in quantitative health services and delivery research; to examine current practice in detecting/mitigating this bias in health services and delivery research systematic reviews; and to explore stakeholders’ perception and experiences concerning such bias.

Methods

The project included five distinct but interrelated work packages. Work package 1 was a systematic review of empirical and methodological studies. Work package 2 involved a survey (meta-epidemiological study) of randomly selected systematic reviews of health services and delivery research topics (n = 200) to evaluate current practice in the assessment of publication and outcome reporting bias during evidence synthesis. Work package 3 included four case studies to explore the applicability of statistical methods for detecting such bias in health services and delivery research. In work package 4 we followed up four cohorts of health services and delivery research studies (total n = 300) to ascertain their publication status, and examined whether publication status was associated with statistical significance or perceived ‘positivity’ of study findings. Work package 5 involved key informant interviews with diverse health services and delivery research stakeholders (n = 24), and a focus group discussion with patient and service user representatives (n = 8).

Results

We identified only four studies that set out to investigate publication and related bias in health services and delivery research in work package 1. Three of these studies focused on health informatics research and one concerned health economics. All four studies reported evidence of the existence of this bias, but had methodological weaknesses. We also identified three health services and delivery research systematic reviews in which findings were compared between published and grey/unpublished literature. These reviews found that the quality and volume of evidence and effect estimates sometimes differed significantly between published and unpublished literature. Work package 2 showed low prevalence of considering/assessing publication (43%) and outcome reporting (17%) bias in health services and delivery research systematic reviews. The prevalence was lower among reviews of associations than among reviews of interventions. The case studies in work package 3 highlighted limitations in current methods for detecting these biases due to heterogeneity and potential confounders. Follow-up of health services and delivery research cohorts in work package 4 showed positive association between publication status and having statistically significant or positive findings. Diverse views concerning publication and related bias and insights into how features of health services and delivery research might influence its occurrence were uncovered through the interviews with health services and delivery research stakeholders and focus group discussion conducted in work package 5.

Conclusions

This study provided prima facie evidence on publication and related bias in quantitative health services and delivery research. This bias does appear to exist, but its prevalence and impact may vary depending on study characteristics, such as study design, and motivation for conducting the evaluation. Emphasis on methodological novelty and focus beyond summative assessments may mitigate/lessen the risk of such bias in health services and delivery research. Methodological and epistemological diversity in health services and delivery research and changing landscape in research publication need to be considered when interpreting the evidence. Collection of further empirical evidence and exploration of optimal health services and delivery research practice are required.

Study registration

This study is registered as PROSPERO CRD42016052333 and CRD42016052366.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 33. See the NIHR Journals Library website for further project information.

Publication and related biases in health services research: a systematic review of empirical evidence

Background

Publication and related biases (including publication bias, time-lag bias, outcome reporting bias and p-hacking) have been well documented in clinical research, but relatively little is known about their presence and extent in health services research (HSR). This paper aims to systematically review evidence concerning publication and related bias in quantitative HSR.

Methods

Databases including MEDLINE, EMBASE, HMIC, CINAHL, Web of Science, Health Systems Evidence, Cochrane EPOC Review Group and several websites were searched to July 2018. Information was obtained from: (1) Methodological studies that set out to investigate publication and related biases in HSR; (2) Systematic reviews of HSR topics which examined such biases as part of the review process. Relevant information was extracted from included studies by one reviewer and checked by another. Studies were appraised according to commonly accepted scientific principles due to lack of suitable checklists. Data were synthesised narratively.

Results

After screening 6155 citations, four methodological studies investigating publication bias in HSR and 184 systematic reviews of HSR topics (including three comparing published with unpublished evidence) were examined. Evidence suggestive of publication bias was reported in some of the methodological studies, but evidence presented was very weak, limited in both quality and scope. Reliable data on outcome reporting bias and p-hacking were scant. HSR systematic reviews in which published literature was compared with unpublished evidence found significant differences in the estimated intervention effects or association in some but not all cases.

Conclusions

Methodological research on publication and related biases in HSR is sparse. Evidence from available literature suggests that such biases may exist in HSR but their scale and impact are difficult to estimate for various reasons discussed in this paper.

Systematic review registration

PROSPERO 2016 CRD42016052333.

Randomised controlled trial of a theory-based intervention to prompt front-line staff to take up the seasonal influenza vaccine

OBJECTIVE

To evaluate the effectiveness of reminder letters informed by social normative theory (a type of 'nudge theory') on uptake of seasonal influenza vaccination by front-line hospital staff.

DESIGN

Individually randomised controlled trial.

SETTING

A large acute care hospital in England.

PARTICIPANTS

Front-line staff employed by the hospital (n=7540) were randomly allocated to one of four reminder types in a factorial design.

INTERVENTIONS

The standard letter included only general information directing the staff to take up the vaccine. A second letter highlighted a type of social norm based on peer comparisons. A third letter highlighted a type of social norm based on an appeal to authority. A fourth letter included a combination of the social norms.

MAIN OUTCOME MEASURE

The proportion of hospital staff vaccinated on-site.

RESULTS

Vaccine coverage was 43% (812/1885) in the standard letter group, 43% (818/1885) in the descriptive norms group, 43% (814/1885) in the injunctive norms group and 43% (812/1885) in the combination group. There were no statistically significant effects of either norm or the interaction. The OR for the descriptive norms factor is 1.01 (0.89-1.15) in the absence of the injunctive norms factor and 1.00 (0.88-1.13) in its presence. The OR for the injunctive norms factor is 1.00 (0.88-1.14) in the absence of the descriptive norms factor and 0.99 (0.87-1.12) in its presence.

CONCLUSIONS

We find no evidence that the uptake of the seasonal influenza vaccination is affected by reminders using social norms to motivate uptake.

Randomised controlled trial of a theory-based intervention to prompt front-line staff to take up the seasonal influenza vaccine

Objective

To evaluate the effectiveness of reminder letters informed by social normative theory (a type of ‘nudge theory’) on uptake of seasonal influenza vaccination by front-line hospital staff.

Design

Individually randomised controlled trial.

Setting

A large acute care hospital in England.

Participants

Front-line staff employed by the hospital (n=7540) were randomly allocated to one of four reminder types in a factorial design.

Interventions

The standard letter included only general information directing the staff to take up the vaccine. A second letter highlighted a type of social norm based on peer comparisons. A third letter highlighted a type of social norm based on an appeal to authority. A fourth letter included a combination of the social norms.

Main outcome measure

The proportion of hospital staff vaccinated on-site.

Results

Vaccine coverage was 43% (812/1885) in the standard letter group, 43% (818/1885) in the descriptive norms group, 43% (814/1885) in the injunctive norms group and 43% (812/1885) in the combination group. There were no statistically significant effects of either norm or the interaction. The OR for the descriptive norms factor is 1.01 (0.89–1.15) in the absence of the injunctive norms factor and 1.00 (0.88–1.13) in its presence. The OR for the injunctive norms factor is 1.00 (0.88–1.14) in the absence of the descriptive norms factor and 0.99 (0.87–1.12) in its presence.

Conclusions

We find no evidence that the uptake of the seasonal influenza vaccination is affected by reminders using social norms to motivate uptake.

Assessment of Health Information Technology Interventions in Evidence-Based Medicine: A Systematic Review by Adopting a Methodological Evaluation Framework

BACKGROUND

The application of Health Information Technologies (HITs) can be an effective way to advance medical research and health services provision. The two-fold objective of this work is to: (i) identify and review state-of-the-art HITs that facilitate the aims of evidence-based medicine and (ii) propose a methodology for HIT assessment.

METHODS

The systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Furthermore, we consolidated existing knowledge in the field and proposed a Synthesis Framework for the Assessment of Health Information Technology (SF/HIT) in order to evaluate the joint use of Randomized Controlled Trials (RCTs) along with HITs in the field of evidence-based medicine.

RESULTS

55 articles met the inclusion criteria and refer to 51 (RCTs) published between 2008 and 2016. Significant improvements in healthcare through the use of HITs were observed in the findings of 31 out of 51 trials-60.8%. We also confirmed that RCTs are valuable tools for assessing the effectiveness, acceptability, safety, privacy, appropriateness, satisfaction, performance, usefulness and adherence.

CONCLUSIONS

To improve health service delivery, RCTs apply and exhibit formalization by providing measurable outputs. Towards this direction, we propose the SF/HIT as a framework which may help researchers to carry out appropriate evaluations and extend their studies.

Evidence-based Health Informatics: How Do We Know What We Know?

Background: Health IT is expected to have a positive impact on the quality and efficiency of health care. But reports on negative impact and patient harm continue to emerge. The obligation of health informatics is to make sure that health IT solutions provide as much benefit with as few negative side effects as possible. To achieve this, health informatics as a discipline must be able to learn, both from its successes as well as from its failures.

Objectives: To present motivation, vision, and history of evidence-based health informatics, and to discuss achievements, challenges, and needs for action.

Methods: Reflections on scientific literature and on own experiences.

Results: Eight challenges on the way towards evidence-based health informatics are identified and discussed: quality of studies; publication bias; reporting quality; availability of publications; systematic reviews and meta-analysis; training of health IT evaluation experts; translation of evidence into health practice; and post-market surveil-lance. Identified needs for action comprise: establish health IT study registers; increase the quality of publications; develop a taxonomy for health IT systems; improve indexing of published health IT evaluation papers; move from meta-analysis to meta-summaries; include health IT evaluation competencies in curricula; develop evidence-based implementation frameworks; and establish post-marketing surveillance for health IT.

Conclusions: There has been some progress, but evidence-based health informatics is still in its infancy. Building evidence in health informatics is our obligation if we consider medical informatics a scientific discipline.

Public Health, Population Health, and Epidemiology Informatics: Recent Research and Trends in the United States

Objectives: To survey advances in public and population health and epidemiology informatics over the past 18 months. Methods: We conducted a review of English-language research works conducted in the domain of public and population health informatics and published in MEDLINE or Web of Science between January 2015 and June 2016 where information technology or informatics was a primary subject or main component of the study methodology. Selected articles were presented using a thematic analysis based on the 2011 American Medical Informatics Association (AMIA) Public Health Informatics Agenda tracks as a typology. Results: Results are given within the context developed by Dixon et al., (2015) and key themes from the 2011 AMIA Public Health Informatics Agenda. Advances are presented within a socio-technical infrastructure undergirded by a trained, competent public health workforce, systems development to meet the business needs of the practice field, and research that evaluates whether those needs are adequately met. The ability to support and grow the infrastructure depends on financial sustainability. Conclusions: The fields of public health and population health informatics continue to grow, with the most notable developments focused on surveillance, workforce development, and linking to or providing clinical services, which encompassed population health informatics advances. Very few advances addressed the need to improve communication, coordination, and consistency with the field of informatics itself, as identified in the AMIA agenda. This will likely result in the persistence of the silos of public health information systems that currently exist. Future research activities need to aim toward a holistic approach of informatics across the enterprise.

Why are medical and health-related studies not being published? A systematic review of reasons given by investigators

Objective

About half of medical and health-related studies are not published. We conducted a systematic review of reports on reasons given by investigators for not publishing their studies in peer-reviewed journals.

Methods

MEDLINE, EMBASE, PsycINFO, and SCOPUS (until 13/09/2013), and references of identified articles were searched to identify reports of surveys that provided data on reasons given by investigators for not publishing studies. The proportion of non-submission and reasons for non-publication was calculated using the number of unpublished studies as the denominator. Because of heterogeneity across studies, quantitative pooling was not conducted. Exploratory subgroup analyses were conducted.

Results

We included 54 survey reports. Data from 38 included reports were available to estimate proportions of at least one reason given for not publishing studies. The proportion of non-submission among unpublished studies ranged from 55% to 100%, with a median of 85%. The reasons given by investigators for not publishing their studies included: lack of time or low priority (median 33%), studies being incomplete (median 15%), study not for publication (median 14%), manuscript in preparation or under review (median 12%), unimportant or negative result (median 12%), poor study quality or design (median 11%), fear of rejection (median 12%), rejection by journals (median 6%), author or co-author problems (median 10%), and sponsor or funder problems (median 9%). In general, the frequency of reasons given for non-publication was not associated with the source of unpublished studies, study design, or time when a survey was conducted.

Conclusions

Non-submission of studies for publication remains the main cause of non-publication of studies. Measures to reduce non-publication of studies and alternative models of research dissemination need to be developed to address the main reasons given by investigators for not publishing their studies, such as lack of time or low priority and fear of being rejected by journals.

Foundation of evidence-based decision making for health care managers, part 1: systematic review

Evidence-based decision making has become a benchmark of best practice. Sources of evidence are systematic reviews and meta-analyses. To support their decision making, health care managers and supervisors need to be able to critically read and interpret systematic reviews and meta-analyses. They also need to be able to determine the applicability of the evidence to their problems and settings. This 2-part series of articles aims to equip health care managers and supervisors with these skills. This article, part 1, explains the types of systematic reviews, defines key terms, and outlines the process of systematic reviews and meta-analyses. Part 2 focuses on the additional procedures associated with meta-analyses, describes the potential shortcomings of both systematic reviews and meta-analyses, and finally, provides a way to appraise the applicability of their results.

Publication bias in clinical trials of electronic health records

OBJECTIVE

To measure the rate of non-publication and assess possible publication bias in clinical trials of electronic health records.

METHODS

We searched ClinicalTrials.gov to identify registered clinical trials of electronic health records and searched the biomedical literature and contacted trial investigators to determine whether the results of the trials were published. Publications were judged as positive, negative, or neutral according to the primary outcome.

RESULTS

Seventy-six percent of trials had publications describing trial results; of these, 74% were positive, 21% were neutral, and 4% were negative (harmful). Of unpublished studies for which the investigator responded, 43% were positive, 57% were neutral, and none were negative; the lower rate of positive results was significant (p<0.001).

CONCLUSION

The rate of non-publication in electronic health record studies is similar to that in other biomedical studies. There appears to be a bias toward publication of positive trials in this domain.

The impact of electronic patient portals on patient care: a systematic review of controlled trials

Background

Modern information technology is changing and provides new challenges to health care. The emergence of the Internet and the electronic health record (EHR) has brought new opportunities for patients to play a more active role in his/her care. Although in many countries patients have the right to access their clinical information, access to clinical records electronically is not common. Patient portals consist of provider-tethered applications that allow patients to electronically access health information that are documented and managed by a health care institution. Although patient portals are already being implemented, it is still unclear in which ways these technologies can influence patient care.

Objective

To systematically review the available evidence on the impact of electronic patient portals on patient care.

Methods

A systematic search was conducted using PubMed and other sources to identify controlled experimental or quasi-experimental studies on the impact of patient portals that were published between 1990 and 2011. A total of 1,306 references from all the publication hits were screened, and 13 papers were retrieved for full text analysis.

Results

We identified 5 papers presenting 4 distinct studies. There were no statistically significant changes between intervention and control group in the 2 randomized controlled trials investigating the effect of patient portals on health outcomes. Significant changes in the patient portal group, compared to a control group, could be observed for the following parameters: quicker decrease in office visit rates and slower increase in telephone contacts; increase in number of messages sent; changes of the medication regimen; and better adherence to treatment.

Conclusions

The number of available controlled studies with regard to patient portals is low. Even when patient portals are often discussed as a way to empower patients and improve quality of care, there is insufficient evidence to support this assumption.

Biomedical informatics--a confluence of disciplines?

BACKGROUND

Biomedical informatics is a broad discipline that borrows many methods and techniques from other disciplines.

OBJECTIVE

To reflect a) on the character of biomedical informatics and to determine whether it is multi-disciplinary or inter-disciplinary; b) on the question whether biomedical informatics is more than the sum of its supporting disciplines and c) on the position of biomedical informatics with respect to related disciplines.

METHOD

Inviting an international group of experts in biomedical informatics and related disciplines on the occasion of the 50th anniversary of Methods of Information in Medicine to present their viewpoints.

RESULTS AND CONCLUSIONS

This paper contains the reflections of a number of the invited experts on the character of biomedical informatics. Most of the authors agree that biomedical informatics is an interdisciplinary field of study where researchers with different scientific backgrounds alone or in combination carry out research. Biomedical informatics is a very broad scientific field and still expanding, yet comprised of a constructive aspect (designing and building systems). One author expressed that the essence of biomedical informatics, as opposed to related disciplines, lies in the modelling of the biomedical content. Interdisciplinarity also has consequences for education. Maintaining rigid disciplinary structures does not allow for sufficient adaptability to capitalize on important trends nor to leverage the influences these trends may have on biomedical informatics. It is therefore important for students to become aware of research findings in related disciplines. In this respect, it was also noted that the fact that many scientific fields use different languages and that the research findings are stored in separate bibliographic databases makes it possible that potentially connected findings will never be linked, despite the fact that these findings were published. Bridges between the sciences are needed for the success of biomedical informatics.

The impact of eHealth on the quality and safety of health care: a systematic overview

BACKGROUND

There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.

METHODS AND FINDINGS

We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.

CONCLUSIONS

There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. In addition, there is a lack of robust research on the risks of implementing these technologies and their cost-effectiveness has yet to be demonstrated, despite being frequently promoted by policymakers and "techno-enthusiasts" as if this was a given. In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.

Vision and challenges of Evidence-Based Health Informatics: a case study of a CPOE meta-analysis

OBJECTIVE

To discuss, taking the example of a meta-analysis on computerized physician order entry (CPOE) systems, the special challenges of Evidence-Based Health Informatics, defined as the conscientious, explicit and judicious use of current best evidence when making decisions about introduction and operation of information technology in a given health care setting.

METHODS

We conducted a case study by performing a systematic review and meta-analysis of CPOE studies. We collected and discussed the challenges we addressed and how they could be overcome.

RESULTS

Challenges comprise the correct identification of published health informatics evaluation studies, the low reporting and study quality of studies, the problem of combining evidence from heterogeneous studies, and the problem of publication bias in health informatics.

CONCLUSION

Based on our experiences while conducting the CPOE meta-analysis, we argue that we are still at the beginning of Evidence-Based Health Informatics. To overcome the discussed challenges, health informatics should strive for harmonized terminology, a study registry, reporting standards, financial or legal incentives for conducting studies, methods to combine evidence from quantitative and qualitative studies, and guidelines for conducting and publishing evaluation studies.

The effect of electronic prescribing on medication errors and adverse drug events: a systematic review

The objective of this systematic review is to analyse the relative risk reduction on medication error and adverse drug events (ADE) by computerized physician order entry systems (CPOE). We included controlled field studies and pretest-posttest studies, evaluating all types of CPOE systems, drugs and clinical settings. We present the results in evidence tables, calculate the risk ratio with 95% confidence interval and perform subgroup analyses for categorical factors, such as the level of care, patient group, type of drug, type of system, functionality of the system, comparison group type, study design, and the method for detecting errors. Of the 25 studies that analysed the effects on the medication error rate, 23 showed a significant relative risk reduction of 13% to 99%. Six of the nine studies that analysed the effects on potential ADEs showed a significant relative risk reduction of 35% to 98%. Four of the seven studies that analysed the effect on ADEs showed a significant relative risk reduction of 30% to 84%. Reporting quality and study quality was often insufficient to exclude major sources of bias. Studies on home-grown systems, studies comparing electronic prescribing to handwriting prescribing, and studies using manual chart review to detect errors seem to show a higher relative risk reduction than other studies. Concluding, it seems that electronic prescribing can reduce the risk for medication errors and ADE. However, studies differ substantially in their setting, design, quality, and results. To further improve the evidence-base of health informatics, more randomized controlled trials (RCTs) are needed, especially to cover a wider range of clinical and geographic settings. In addition, reporting quality of health informatics evaluation studies has to be substantially improved.