Converge or Collide? Making Sense of a Plethora of Open Data Standards in Health Care

Practitioners of digital health are familiar with disjointed data environments that often inhibit effective communication among different elements of the ecosystem. This fragmentation leads in turn to issues such as inconsistencies in services versus payments, wastage, and notably, care delivered being less than best-practice. Despite the long-standing recognition of interoperable data as a potential solution, efforts in achieving interoperability have been disjointed and inconsistent, resulting in numerous incompatible standards, despite the widespread agreement that fewer standards would enhance interoperability. This paper introduces a framework for understanding health care data needs, discussing the challenges and opportunities of open data standards in the field. It emphasizes the necessity of acknowledging diverse data standards, each catering to specific viewpoints and needs, while proposing a categorization of health care data into three domains, each with its distinct characteristics and challenges, along with outlining overarching design requirements applicable to all domains and specific requirements unique to each domain.

A computable biomedical knowledge object for calculating in-hospital mortality for patients admitted with acute myocardial infarction

Introduction

Quality indicators play an essential role in a learning health system. They help healthcare providers to monitor the quality and safety of care delivered and to identify areas for improvement. Clinical quality indicators, therefore, need to be based on real world data. Generating reliable and actionable data routinely is challenging. Healthcare data are often stored in different formats and use different terminologies and coding systems, making it difficult to generate and compare indicator reports from different sources.

Methods

The Observational Health Sciences and Informatics community maintains the Observational Medical Outcomes Partnership Common Data Model (OMOP). This is an open data standard providing a computable and interoperable format for real world data. We implemented a Computable Biomedical Knowledge Object (CBK) in the Piano Platform based on OMOP. The CBK calculates an inpatient quality indicator and was illustrated using synthetic electronic health record (EHR) data in the open OMOP standard.

Results

The CBK reported the in-hospital mortality of patients admitted for acute myocardial infarction (AMI) for the synthetic EHR dataset and includes interactive visualizations and the results of calculations. Value sets composed of OMOP concept codes for AMI and comorbidities used in the indicator calculation were also created.

Conclusion

Computable biomedical knowledge (CBK) objects that operate on OMOP data can be reused across datasets that conform to OMOP. With OMOP being a widely used interoperability standard, quality indicators embedded in CBKs can accelerate the generation of evidence for targeted quality and safety management, improving care to benefit larger populations.

Evidence Hub: A place to exchange medical knowledge and form communities

Introduction

Medical knowledge is complex and constantly evolving, making it challenging to disseminate and retrieve effectively. To address these challenges, researchers are exploring the use of formal knowledge representations that can be easily interpreted by computers.

Methods

Evidence Hub is a new, free, online platform that hosts computable clinical knowledge in the form of "Knowledge Objects". These objects represent various types of computer-interpretable knowledge. The platform includes features that encourage advancing medical knowledge, such as public discussion threads for civil discourse about each Knowledge Object, thus building communities of interest that can form and reach consensus on the correctness, applicability, and proper use of the object. Knowledge Objects are maintained by volunteers and published on Evidence Hub under GPL 2.0. Peer review and quality assurance are provided by volunteers.

Results

Users can explore Evidence Hub and participate in discussions using a web browser. An application programming interface allows applications to register themselves as handlers of specific object types and provide editing and execution capabilities for particular object types.

Conclusions

By providing a platform for computable clinical knowledge and fostering discussion and collaboration, Evidence Hub improves the dissemination and use of medical knowledge.

A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?

BACKGROUND

Although many aspects of systematic reviews use computational tools, systematic reviewers have been reluctant to adopt machine learning tools.

DISCUSSION

We discuss that the potential reason for the slow adoption of machine learning tools into systematic reviews is multifactorial. We focus on the current absence of trust in automation and set-up challenges as major barriers to adoption. It is important that reviews produced using automation tools are considered non-inferior or superior to current practice. However, this standard will likely not be sufficient to lead to widespread adoption. As with many technologies, it is important that reviewers see "others" in the review community using automation tools. Adoption will also be slow if the automation tools are not compatible with workflows and tasks currently used to produce reviews. Many automation tools being developed for systematic reviews mimic classification problems. Therefore, the evidence that these automation tools are non-inferior or superior can be presented using methods similar to diagnostic test evaluations, i.e., precision and recall compared to a human reviewer. However, the assessment of automation tools does present unique challenges for investigators and systematic reviewers, including the need to clarify which metrics are of interest to the systematic review community and the unique documentation challenges for reproducible software experiments.

CONCLUSION

We discuss adoption barriers with the goal of providing tool developers with guidance as to how to design and report such evaluations and for end users to assess their validity. Further, we discuss approaches to formatting and announcing publicly available datasets suitable for assessment of automation technologies and tools. Making these resources available will increase trust that tools are non-inferior or superior to current practice. Finally, we identify that, even with evidence that automation tools are non-inferior or superior to current practice, substantial set-up challenges remain for main stream integration of automation into the systematic review process.

Automated annotation of mobile antibiotic resistance in Gram-negative bacteria: the Multiple Antibiotic Resistance Annotator (MARA) and database

Background

Multiresistance in Gram-negative bacteria is often due to acquisition of several different antibiotic resistance genes, each associated with a different mobile genetic element, that tend to cluster together in complex conglomerations. Accurate, consistent annotation of resistance genes, the boundaries and fragments of mobile elements, and signatures of insertion, such as DR, facilitates comparative analysis of complex multiresistance regions and plasmids to better understand their evolution and how resistance genes spread.

Objectives

To extend the Repository of Antibiotic resistance Cassettes (RAC) web site, which includes a database of 'features', and the Attacca automatic DNA annotation system, to encompass additional resistance genes and all types of associated mobile elements.

Methods

Antibiotic resistance genes and mobile elements were added to RAC, from existing registries where possible. Attacca grammars were extended to accommodate the expanded database, to allow overlapping features to be annotated and to identify and annotate features such as composite transposons and DR.

Results

The Multiple Antibiotic Resistance Annotator (MARA) database includes antibiotic resistance genes and selected mobile elements from Gram-negative bacteria, distinguishing important variants. Sequences can be submitted to the MARA web site for annotation. A list of positions and orientations of annotated features, indicating those that are truncated, DR and potential composite transposons is provided for each sequence, as well as a diagram showing annotated features approximately to scale.

Conclusions

The MARA web site (http://mara.spokade.com) provides a comprehensive database for mobile antibiotic resistance in Gram-negative bacteria and accurately annotates resistance genes and associated mobile elements in submitted sequences to facilitate comparative analysis.

Still moving toward automation of the systematic review process: a summary of discussions at the third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR)

The third meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 17-18 October 2017 in London, England. ICASR is an interdisciplinary group whose goal is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. The group seeks to facilitate the development and widespread acceptance of automated techniques for systematic reviews. The meeting's conclusion was that the most pressing needs at present are to develop approaches for validating currently available tools and to provide increased access to curated corpora that can be used for validation. To that end, ICASR's short-term goals in 2018-2019 are to propose and publish protocols for key tasks in systematic reviews and to develop an approach for sharing curated corpora for validating the automation of the key tasks.

Ecological effects of cefepime use during antibiotic cycling on the Gram-negative enteric flora of ICU patients

This study examines the impact of cefepime and APP-β (antipseudomonal penicillin/ β-lactamase inhibitor combinations) on Gram-negative bacterial colonization and resistance in two Australian ICUs. While resistance did not cumulatively increase, cefepime (but not APP-β treatment) was associated with acquisition of antibiotic resistant Enterobacteriaceae, consistent with an ecological effect. Analysis of the resident gut E. coli population in a subset of patients showed an increase in markers of horizontal gene transfer after cefepime exposure that helps explain the increase in APP-β resistance and reminds us that unmeasured impacts on the microbiome are key outcome determinants that need to be fully explored.

Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR)

Systematic reviews (SR) are vital to health care, but have become complicated and time-consuming, due to the rapid expansion of evidence to be synthesised. Fortunately, many tasks of systematic reviews have the potential to be automated or may be assisted by automation. Recent advances in natural language processing, text mining and machine learning have produced new algorithms that can accurately mimic human endeavour in systematic review activity, faster and more cheaply. Automation tools need to be able to work together, to exchange data and results. Therefore, we initiated the International Collaboration for the Automation of Systematic Reviews (ICASR), to successfully put all the parts of automation of systematic review production together. The first meeting was held in Vienna in October 2015. We established a set of principles to enable tools to be developed and integrated into toolkits.This paper sets out the principles devised at that meeting, which cover the need for improvement in efficiency of SR tasks, automation across the spectrum of SR tasks, continuous improvement, adherence to high quality standards, flexibility of use and combining components, the need for a collaboration and varied skills, the desire for open source, shared code and evaluation, and a requirement for replicability through rigorous and open evaluation.Automation has a great potential to improve the speed of systematic reviews. Considerable work is already being done on many of the steps involved in a review. The 'Vienna Principles' set out in this paper aim to guide a more coordinated effort which will allow the integration of work by separate teams and build on the experience, code and evaluations done by the many teams working across the globe.

Automated screening of research studies for systematic reviews using study characteristics

BACKGROUND

Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%.

METHODS

We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews.

RESULTS

The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%.

CONCLUSIONS

Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.

Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR)

The second meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 3-4 October 2016 in Philadelphia, Pennsylvania, USA. ICASR is an interdisciplinary group whose aim is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. Having automated tools for systematic review should enable more transparent and timely review, maximizing the potential for identifying and translating research findings to practical application. The meeting brought together multiple stakeholder groups including users of summarized research, methodologists who explore production processes and systematic review quality, and technologists such as software developers, statisticians, and vendors. This diversity of participants was intended to ensure effective communication with numerous stakeholders about progress toward automation of systematic reviews and stimulate discussion about potential solutions to identified challenges. The meeting highlighted challenges, both simple and complex, and raised awareness among participants about ongoing efforts by various stakeholders. An outcome of this forum was to identify several short-term projects that participants felt would advance the automation of tasks in the systematic review workflow including (1) fostering better understanding about available tools, (2) developing validated datasets for testing new tools, (3) determining a standard method to facilitate interoperability of tools such as through an application programming interface or API, and (4) establishing criteria to evaluate the quality of tools' output. ICASR 2016 provided a beneficial forum to foster focused discussion about tool development and resources and reconfirm ICASR members' commitment toward systematic reviews' automation.

Linking quality indicators to clinical trials: an automated approach

OBJECTIVE

Quality improvement of health care requires robust measurable indicators to track performance. However identifying which indicators are supported by strong clinical evidence, typically from clinical trials, is often laborious. This study tests a novel method for automatically linking indicators to clinical trial registrations.

DESIGN

A set of 522 quality of care indicators for 22 common conditions drawn from the CareTrack study were automatically mapped to outcome measures reported in 13 971 trials from ClinicalTrials.gov.

INTERVENTION

Text mining methods extracted phrases mentioning indicators and outcome phrases, and these were compared using the Levenshtein edit distance ratio to measure similarity.

MAIN OUTCOME MEASURE

Number of care indicators that mapped to outcome measures in clinical trials.

RESULTS

While only 13% of the 522 CareTrack indicators were thought to have Level I or II evidence behind them, 353 (68%) could be directly linked to randomized controlled trials. Within these 522, 50 of 70 (71%) Level I and II evidence-based indicators, and 268 of 370 (72%) Level V (consensus-based) indicators could be linked to evidence. Of the indicators known to have evidence behind them, only 5.7% (4 of 70) were mentioned in the trial reports but were missed by our method.

CONCLUSIONS

We automatically linked indicators to clinical trial registrations with high precision. Whilst the majority of quality indicators studied could be directly linked to research evidence, a small portion could not and these require closer scrutiny. It is feasible to support the process of indicator development using automated methods to identify research evidence.

Linking clinical quality indicators to research evidence - a case study in asthma management for children

BACKGROUND

Clinical quality indicators are used to monitor the performance of healthcare services and should wherever possible be based on research evidence. Little is known however about the extent to which indicators in common use are based on research. The objective of this study is to measure the extent to which clinical quality indicators used in asthma management in children with outcome measurements can be linked to results in randomised controlled clinical trial (RCT) reports. This work is part of a broader research program to trial methods that improve the efficiency and accuracy of indicator development.

METHODS

National-level indicators for asthma management in children were extracted from the National Quality Measures Clearinghouse database and the National Institute for Health and Care Excellence quality standards by two independent appraisers. Outcome measures were extracted from all published English language RCT reports for asthma management in children below the age of 12 published between 2005 and 2014. The two sets were then linked by manually mapping both to a common set of Unified Medical Language System (UMLS) concepts.

RESULTS

The analysis identified 39 indicators and 562 full text RCTs dealing with asthma management in children. About 95% (37/39) of the indicators could be linked to RCT outcome measures.

CONCLUSIONS

It is possible to identify relevant RCT reports for the majority of indicators used to assess the quality of asthma management in childhood. The methods reported here could be automated to more generally support assessment of candidate indicators against the research evidence.

Evaluation of a rule-based method for epidemiological document classification towards the automation of systematic reviews

INTRODUCTION

Most data extraction efforts in epidemiology are focused on obtaining targeted information from clinical trials. In contrast, limited research has been conducted on the identification of information from observational studies, a major source for human evidence in many fields, including environmental health. The recognition of key epidemiological information (e.g., exposures) through text mining techniques can assist in the automation of systematic reviews and other evidence summaries.

METHOD

We designed and applied a knowledge-driven, rule-based approach to identify targeted information (study design, participant population, exposure, outcome, confounding factors, and the country where the study was conducted) from abstracts of epidemiological studies included in several systematic reviews of environmental health exposures. The rules were based on common syntactical patterns observed in text and are thus not specific to any systematic review. To validate the general applicability of our approach, we compared the data extracted using our approach versus hand curation for 35 epidemiological study abstracts manually selected for inclusion in two systematic reviews.

RESULTS

The returned F-score, precision, and recall ranged from 70% to 98%, 81% to 100%, and 54% to 97%, respectively. The highest precision was observed for exposure, outcome and population (100%) while recall was best for exposure and study design with 97% and 89%, respectively. The lowest recall was observed for the population (54%), which also had the lowest F-score (70%).

CONCLUSION

The generated performance of our text-mining approach demonstrated encouraging results for the identification of targeted information from observational epidemiological study abstracts related to environmental exposures. We have demonstrated that rules based on generic syntactic patterns in one corpus can be applied to other observational study design by simple interchanging the dictionaries aiming to identify certain characteristics (i.e., outcomes, exposures). At the document level, the recognised information can assist in the selection and categorization of studies included in a systematic review.

Measuring the effects of computer downtime on hospital pathology processes

OBJECTIVE

To introduce and evaluate a method that uses electronic medical record (EMR) data to measure the effects of computer system downtime on clinical processes associated with pathology testing and results reporting.

MATERIALS AND METHODS

A matched case-control design was used to examine the effects of five downtime events over 11-months, ranging from 5 to 300min. Four indicator tests representing different laboratory workflows were selected to measure delays and errors: potassium, haemoglobon, troponin and activated partial thromboplastin time. Tests exposed to a downtime were matched to tests during unaffected control periods by test type, time of day and day of week. Measures included clinician read time (CRT), laboratory turnaround time (LTAT), and rates of missed reads, futile searches, duplicate orders, and missing test results.

RESULTS

The effects of downtime varied with the type of IT problem. When clinicians could not logon to a results reporting system for 17-min, the CRT for potassium and haemoglobon tests was five (10.3 vs. 2.0days) and six times (13.4 vs. 2.1days) longer than control (p=0.01-0.04; p=0.0001-0.003). Clinician follow-up of tests was also delayed by another downtime involving a power outage with a small effect. In contrast, laboratory processing of troponin tests was unaffected by network services and routing problems. Errors including missed reads, futile searches, duplicate orders and missing test results could not be examined because the sample size of affected tests was not sufficient for statistical testing.

CONCLUSION

This study demonstrates the feasibility of using routinely collected EMR data with a matched case-control design to measure the effects of downtime on clinical processes. Even brief system downtimes may impact patient care. The methodology has potential to be applied to other clinical processes with established workflows where tasks are pre-defined such as medications management.

Comparing clinical quality indicators for asthma management in children with outcome measures used in randomised controlled trials: a protocol

INTRODUCTION

Clinical quality indicators are necessary to monitor the performance of healthcare services. The development of indicators should, wherever possible, be based on research evidence to minimise the risk of bias which may be introduced during their development, because of logistic, ethical or financial constraints alone. The development of automated methods to identify the evidence base for candidate indicators should improve the process of indicator development. The objective of this study is to explore the relationship between clinical quality indicators for asthma management in children with outcome and process measurements extracted from randomised controlled clinical trial reports.

METHODS AND ANALYSIS

National-level indicators for asthma management in children will be extracted from the National Quality Measures Clearinghouse (NQMC) database and the National Institute for Health and Care Excellence (NICE) quality standards. Outcome measures will be extracted from published English language randomised controlled trial (RCT) reports for asthma management in children aged below 12 years. The two sets of measures will be compared to assess any overlap. The study will provide insights into the relationship between clinical quality indicators and measurements in RCTs. This study will also yield a list of measurements used in RCTs for asthma management in children, and will find RCT evidence for indicators used in practice.

ETHICS AND DISSEMINATION

Ethical approval is not necessary because this study will not include patient data. Findings will be disseminated through peer-reviewed publications.

Using social connection information to improve opinion mining: Identifying negative sentiment about HPV vaccines on Twitter

The manner in which people preferentially interact with others like themselves suggests that information about social connections may be useful in the surveillance of opinions for public health purposes. We examined if social connection information from tweets about human papillomavirus (HPV) vaccines could be used to train classifiers that identify anti-vaccine opinions. From 42,533 tweets posted between October 2013 and March 2014, 2,098 were sampled at random and two investigators independently identified anti-vaccine opinions. Machine learning methods were used to train classifiers using the first three months of data, including content (8,261 text fragments) and social connections (10,758 relationships). Connection-based classifiers performed similarly to content-based classifiers on the first three months of training data, and performed more consistently than content-based classifiers on test data from the subsequent three months. The most accurate classifier achieved an accuracy of 88.6% on the test data set, and used only social connection features. Information about how people are connected, rather than what they write, may be useful for improving public health surveillance methods on Twitter.

Financial conflicts of interest and conclusions about neuraminidase inhibitors for influenza: an analysis of systematic reviews

BACKGROUND

Industry funding and financial conflicts of interest may contribute to bias in the synthesis and interpretation of scientific evidence.

OBJECTIVE

To examine the association between financial conflicts of interest and characteristics of systematic reviews of neuraminidase inhibitors.

DESIGN

Retrospective analysis.

SETTING

Reviews that examined the use of neuraminidase inhibitors in the prophylaxis or treatment of influenza, were published between January 2005 and May 2014, and used a systematic search protocol.

MEASUREMENTS

Two investigators blinded to all information regarding the review authors independently assessed the presentation of evidence on the use of neuraminidase inhibitors as favorable or not favorable. Financial conflicts of interest were identified using the index reviews, other publications, and Web-based searches. Associations between financial conflicts of interest, favorability assessments, and presence of critical appraisals of evidence quality were analyzed.

RESULTS

Twenty-six systematic reviews were identified, of which 13 examined prophylaxis and 24 examined treatment, accounting for 37 distinct assessments. Among assessments associated with a financial conflict of interest, 7 of 8 (88%) were classified as favorable, compared with 5 of 29 (17%) among those without a financial conflict of interest. Reviewers without financial conflicts of interest were more likely to include statements about the quality of the primary studies than those with financial conflicts of interest.

LIMITATIONS

The heterogeneity in populations and outcomes examined in the reviews precluded analysis of the contribution of selective inclusion of evidence on the discordance of the assessments made in the reviews. Many of the systematic reviews had overlapping authorship.

CONCLUSION

Reviewers with financial conflicts of interest may be more likely to present evidence about neuraminidase inhibitors in a favorable manner and recommend the use of these drugs than reviewers without financial conflicts of interest.

PRIMARY FUNDING SOURCE

Australian National Health and Medical Research Council.

Automatic evidence retrieval for systematic reviews

Background

Snowballing involves recursively pursuing relevant references cited in the retrieved literature and adding them to the search results. Snowballing is an alternative approach to discover additional evidence that was not retrieved through conventional search. Snowballing’s effectiveness makes it best practice in systematic reviews despite being time-consuming and tedious.

Objective

Our goal was to evaluate an automatic method for citation snowballing’s capacity to identify and retrieve the full text and/or abstracts of cited articles.

Methods

Using 20 review articles that contained 949 citations to journal or conference articles, we manually searched Microsoft Academic Search (MAS) and identified 78.0% (740/949) of the cited articles that were present in the database. We compared the performance of the automatic citation snowballing method against the results of this manual search, measuring precision, recall, and F₁ score.

Results

The automatic method was able to correctly identify 633 (as proportion of included citations: recall=66.7%, F₁ score=79.3%; as proportion of citations in MAS: recall=85.5%, F₁ score=91.2%) of citations with high precision (97.7%), and retrieved the full text or abstract for 490 (recall=82.9%, precision=92.1%, F₁ score=87.3%) of the 633 correctly retrieved citations.

Conclusions

The proposed method for automatic citation snowballing is accurate and is capable of obtaining the full texts or abstracts for a substantial proportion of the scholarly citations in review articles. By automating the process of citation snowballing, it may be possible to reduce the time and effort of common evidence surveillance tasks such as keeping trial registries up to date and conducting systematic reviews.

Systematic review automation technologies

Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects.We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.

Citation networks of related trials are often disconnected: implications for bidirectional citation searches

BACKGROUND AND OBJECTIVES

Reports of randomized controlled trials (RCTs) should set findings within the context of previous research. The resulting network of citations would also provide an alternative search method for clinicians, researchers, and systematic reviewers seeking to base decisions on all available evidence. We sought to determine the connectedness of citation networks of RCTs by examining direct (referenced trials) and indirect (through references of referenced trials, etc) citation of trials to one another.

METHODS

Meta-analyses were used to create citation networks of RCTs addressing the same clinical questions. The primary measure was the proportion of networks where following citation links between RCTs identifies the complete set of RCTs, forming a single connected citation group. Other measures included the number of disconnected groups (islands) within each network, the number of citations in the network relative to the maximum possible, and the maximum number of links in the path between two connected trials (a measure of indirectness of citations).

RESULTS

We included 259 meta-analyses with a total of 2,413 and a median of seven RCTs each. For 46% (118 of 259) of networks, the RCTs formed a single connected citation group-one island. For the other 54% of networks, where at least one RCT group was not cited by others, 39% had two citation islands and 4% (10 of 257) had 10 or more islands. On average, the citation networks had 38% of the possible citations to other trials (if each trial had cited all earlier trials). The number of citation islands and the maximum number of citation links increased with increasing numbers of trials in the network.

CONCLUSION

Available evidence to answer a clinical question may be identified by using network citations created with a small initial corpus of eligible trials. However, the number of islands means that citation networks cannot be relied on for evidence retrieval.

Role of citation tracking in updating of systematic reviews

We proposed to use automatic citation tracking to enhance the retrieval of new evidence for updating Systematic Reviews (SR). We tested on a Cochrane review from 2003 (updated 2010) and retrieved 12 of the papers to be added (recall 85.7%). Citation tracking yields a high proportion of the required literature.

Gene-disease association with literature based enrichment

MOTIVATION

Gene set enrichment analysis (GSEA) annotates gene microarray data with functional information from the biomedical literature to improve gene-disease association prediction. We hypothesize that supplementing GSEA with comprehensive gene function catalogs built automatically using information extracted from the scientific literature will significantly enhance GSEA prediction quality.

METHODS

Gold standard gene sets for breast cancer (BrCa) and colorectal cancer (CRC) were derived from the literature. Two gene function catalogs (CMeSH and CUMLS) were automatically generated. 1. By using Entrez Gene to associate all recorded human genes with PubMed article IDs. 2. Using the genes mentioned in each PubMed article and associating each with the article's MeSH terms (in CMeSH) and extracted UMLS concepts (in CUMLS). Microarray data from the Gene Expression Omnibus for BrCa and CRC was then annotated using CMeSH and CUMLS and for comparison, also with several pre-existing catalogs (C2, C4 and C5 from the Molecular Signatures Database). Ranking was done using, a standard GSEA implementation (GSEA-p). Gene function predictions for enriched array data were evaluated against the gold standard by measuring area under the receiver operating characteristic curve (AUC).

RESULTS

Comparison of ranking using the literature enrichment catalogs, the pre-existing catalogs as well as five randomly generated catalogs show the literature derived enrichment catalogs are more effective. The AUC for BrCa using the unenriched gene expression dataset was 0.43, increasing to 0.89 after gene set enrichment with CUMLS. The AUC for CRC using the unenriched gene expression dataset was 0.54, increasing to 0.9 after enrichment with CMeSH. C2 increased AUC (BrCa 0.76, CRC 0.71) but C4 and C5 performed poorly (between 0.35 and 0.5). The randomly generated catalogs also performed poorly, equivalent to random guessing.

DISCUSSION

Gene set enrichment significantly improved prediction of gene-disease association. Selection of enrichment catalog had a substantial effect on prediction accuracy. The literature based catalogs performed better than the MSigDB catalogs, possibly because they are more recent. Catalogs generated automatically from the literature can be kept up to date.

CONCLUSION

Prediction of gene-disease association is a fundamental task in biomedical research. GSEA provides a promising method when using literature-based enrichment catalogs.

AVAILABILITY

The literature based catalogs generated and used in this study are available from http://www2.chi.unsw.edu.au/literature-enrichment.

The implications of biomarker evidence for systematic reviews

BACKGROUND

In Evidence-Based Medicine, clinical practice guidelines and systematic reviews are crucial devices for medical practitioners in making clinical decision. Clinical practice guidelines are systematically developed statements to support health care decisions for specific circumstances whereas systematic reviews are summaries of evidence on clearly formulated clinical questions. Biomarkers are biological measurements (primarily molecular) that are used to diagnose, predict treatment outcomes and prognosticate disease and are increasingly used in randomized controlled trials (RCT).

METHODS

We search PubMed for systematic reviews, RCTs, case reports and non-systematic reviews with and without mentions of biomarkers between years 1990-2011. We compared the frequency and growth rate of biomarkers and non-biomarkers publications. We also compared the growth of the proportion of biomarker-based RCTs with the growth of the proportion of biomarker-based systematic reviews.

RESULTS

With 147,774 systematic reviews indexed in PubMed from 1990 to 2011 (accessed on 18/10/2012), only 4,431 (3%) are dedicated to biomarkers. The annual growth rate of biomarkers publications is consistently higher than non-biomarkers publications, showing the growth in biomarkers research. From 20 years of systematic review publications indexed in PubMed, we identified a bias in systematic reviews against the inclusion of biomarker-based RCTs.

CONCLUSIONS

With the realisation of genome-based personalised medicine, biomarkers are becoming important for clinical decision making. The bias against the inclusion of biomarkers in systematic reviews leads to medical practitioners deprive of important information they require to address clinical questions. Sparse or weak evidence and lack of genetic training for systematic reviewers may contribute to this trend.

Genetic and epigenetic biomarkers of colorectal cancer

Cancer is a heterogeneous disease caused, in part, by genetic and epigenetic alterations. These changes have been explored in studies of the pathogenesis of colorectal cancer (CRC) and have led to the identification of many biomarkers of disease progression. However, the number of biomarkers that have been incorporated into clinical practice is surprisingly small. We review the genetic and epigenetic mechanisms of colorectal cancer and discuss molecular markers recommended for use in early detection, screening, diagnosis, determination of prognosis, and prediction of treatment outcomes. We also review important areas for future research.

RAC: Repository of Antibiotic resistance Cassettes

Antibiotic resistance in bacteria is often due to acquisition of resistance genes associated with different mobile genetic elements. In Gram-negative bacteria, many resistance genes are found as part of small mobile genetic elements called gene cassettes, generally found integrated into larger elements called integrons. Integrons carrying antibiotic resistance gene cassettes are often associated with mobile elements and here are designated ‘mobile resistance integrons’ (MRIs). More than one cassette can be inserted in the same integron to create arrays that contribute to the spread of multi-resistance. In many sequences in databases such as GenBank, only the genes within cassettes, rather than whole cassettes, are annotated and the same gene/cassette may be given different names in different entries, hampering analysis. We have developed the Repository of Antibiotic resistance Cassettes (RAC) website to provide an archive of gene cassettes that includes alternative gene names from multiple nomenclature systems and allows the community to contribute new cassettes. RAC also offers an additional function that allows users to submit sequences containing cassettes or arrays for annotation using the automatic annotation system Attacca. Attacca recognizes features (gene cassettes, integron regions) and identifies cassette arrays as patterns of features and can also distinguish minor cassette variants that may encode different resistance phenotypes (aacA4 cassettes and bla cassettes-encoding β-lactamases). Gaps in annotations are manually reviewed and those found to correspond to novel cassettes are assigned unique names. While there are other websites dedicated to integrons or antibiotic resistance genes, none includes a complete list of antibiotic resistance gene cassettes in MRI or offers consistent annotation and appropriate naming of all of these cassettes in submitted sequences. RAC thus provides a unique resource for researchers, which should reduce confusion and improve the quality of annotations of gene cassettes in integrons associated with antibiotic resistance.

Database URL:http://www2.chi.unsw.edu.au/rac.

BICEPP: an example-based statistical text mining method for predicting the binary characteristics of drugs

Background

The identification of drug characteristics is a clinically important task, but it requires much expert knowledge and consumes substantial resources. We have developed a statistical text-mining approach (BInary Characteristics Extractor and biomedical Properties Predictor: BICEPP) to help experts screen drugs that may have important clinical characteristics of interest.

Results

BICEPP first retrieves MEDLINE abstracts containing drug names, then selects tokens that best predict the list of drugs which represents the characteristic of interest. Machine learning is then used to classify drugs using a document frequency-based measure. Evaluation experiments were performed to validate BICEPP's performance on 484 characteristics of 857 drugs, identified from the Australian Medicines Handbook (AMH) and the PharmacoKinetic Interaction Screening (PKIS) database. Stratified cross-validations revealed that BICEPP was able to classify drugs into all 20 major therapeutic classes (100%) and 157 (of 197) minor drug classes (80%) with areas under the receiver operating characteristic curve (AUC) > 0.80. Similarly, AUC > 0.80 could be obtained in the classification of 173 (of 238) adverse events (73%), up to 12 (of 15) groups of clinically significant cytochrome P450 enzyme (CYP) inducers or inhibitors (80%), and up to 11 (of 14) groups of narrow therapeutic index drugs (79%). Interestingly, it was observed that the keywords used to describe a drug characteristic were not necessarily the most predictive ones for the classification task.

Conclusions

BICEPP has sufficient classification power to automatically distinguish a wide range of clinical properties of drugs. This may be used in pharmacovigilance applications to assist with rapid screening of large drug databases to identify important characteristics for further evaluation.

Computational reasoning across multiple models

Computational support of clinical decisions frequently requires the integration of data in a variety of formats and from multiple sources and domains. Some impressive multiscale computational models of biological phenomena have been developed as part of the study of disease and healthcare systems. One can now contemplate harnessing these models arising from computational biology and using highly interconnected clinical data to support clinical decision-making. Indeed, understanding how to build computational systems able to reason across heterogeneous models and datasets is one of the major and perhaps foundational challenges of translational biomedical informatics. In this paper, the authors examine the use of multimodels (models composed of several daughter models) and explore three major research challenges to reasoning across multiple models: model selection, model composition, and computer aided model construction.

Context-driven discovery of gene cassettes in mobile integrons using a computational grammar

Background

Gene discovery algorithms typically examine sequence data for low level patterns. A novel method to computationally discover higher order DNA structures is presented, using a context sensitive grammar. The algorithm was applied to the discovery of gene cassettes associated with integrons. The discovery and annotation of antibiotic resistance genes in such cassettes is essential for effective monitoring of antibiotic resistance patterns and formulation of public health antibiotic prescription policies.

Results

We discovered two new putative gene cassettes using the method, from 276 integron features and 978 GenBank sequences. The system achieved κ = 0.972 annotation agreement with an expert gold standard of 300 sequences. In rediscovery experiments, we deleted 789,196 cassette instances over 2030 experiments and correctly relabelled 85.6% (α ≥ 95%, E ≤ 1%, mean sensitivity = 0.86, specificity = 1, F-score = 0.93), with no false positives.

Error analysis demonstrated that for 72,338 missed deletions, two adjacent deleted cassettes were labeled as a single cassette, increasing performance to 94.8% (mean sensitivity = 0.92, specificity = 1, F-score = 0.96).

Conclusion

Using grammars we were able to represent heuristic background knowledge about large and complex structures in DNA. Importantly, we were also able to use the context embedded in the model to discover new putative antibiotic resistance gene cassettes. The method is complementary to existing automatic annotation systems which operate at the sequence level.

The Changing Nature of Clinical Decision Support Systems: a Focus on Consumers, Genomics, Public Health and Decision Safety

Objectives To review the recent research literature in clinical decision support systems (CDSS).

Methods A review of recent literature was undertaken, focussing on CDSS evaluation, consumers and public health, the impact of translational bioinformatics on CDSS design, and CDSS safety.

Results In recent years, researchers have concentrated much less on the development of decision technologies, and have focussed more on the impact of CDSS in the clinical world. Recent work highlights that traditional process measures of CDSS effectiveness, such as document relevance are poor proxy measures for decision outcomes. Measuring the dynamics of decision making, for example via decision velocity, may produce a more accurate picture of effectiveness. Another trend is the broadening of user base for CDSS beyond front line clinicians. Consumers are now a major focus for biomedical informatics, as are public health officials, tasked with detecting and managing disease outbreaks at a health system, rather than individual patient level. Bioinformatics is also changing the nature of CDSS. Apart from personalisation of therapy recommendations, translational bioinformatics is creating new challenges in the interpretation of the meaning of genetic data. Finally, there is much recent interest in the safety and effectiveness of computerised physicianorderentry (CPOE) systems, given that prescribing and administration errors are a significant cause of morbidity and mortality. Of note, there is still much controversy surrounding the contention that poorly designed, implemented or used CDSS may actually lead to harm.

Conclusions CDSS research remains an active and evolving area of research, as CDSS penetrate more widely beyond their traditional domain into consumer decision support, and as decisions become more complex, for example by involving sequence level genetic data.

Gene cassettes and cassette arrays in mobile resistance integrons

Gene cassettes are small mobile elements, consisting of little more than a single gene and recombination site, which are captured by larger elements called integrons. Several cassettes may be inserted into the same integron forming a tandem array. The discovery of integrons in the chromosome of many species has led to the identification of thousands of gene cassettes, mostly of unknown function, while integrons associated with transposons and plasmids carry mainly antibiotic resistance genes and constitute an important means of spreading resistance. An updated compilation of gene cassettes found in sequences of such 'mobile resistance integrons' in GenBank was facilitated by a specially developed automated annotation system. At least 130 different (<98% identical) cassettes that carry known or predicted antibiotic resistance genes were identified, along with many cassettes of unknown function. We list exemplar GenBank accession numbers for each and address some nomenclature issues. Various modifications to cassettes, some of which may be useful in tracking cassette epidemiology, are also described. Despite potential biases in the GenBank dataset, preliminary analysis of cassette distribution suggests interesting differences between cassettes and may provide useful information to direct more systematic studies.

The changing nature of clinical decision support systems: a focus on consumers, genomics, public health and decision safety

OBJECTIVES

To review the recent research literature in clinical decision support systems (CDSS).

METHODS

A review of recent literature was undertaken, focussing on CDSS evaluation, consumers and public health, the impact of translational bioinformatics on CDSS design, and CDSS safety.

RESULTS

In recent years, researchers have concentrated much less on the development of decision technologies, and have focussed more on the impact of CDSS in the clinical world. Recent work highlights that traditional process measures of CDSS effectiveness, such as document relevance are poor proxy measures for decision outcomes. Measuring the dynamics of decision making, for example via decision velocity, may produce a more accurate picture of effectiveness. Another trend is the broadening of user base for CDSS beyond front line clinicians. Consumers are now a major focus for biomedical informatics, as are public health officials, tasked with detecting and managing disease outbreaks at a health system, rather than individual patient level. Bioinformatics is also changing the nature of CDSS. Apart from personalisation of therapy recommendations, translational bioinformatics is creating new challenges in the interpretation of the meaning of genetic data. Finally, there is much recent interest in the safety and effectiveness of computerised physician order entry (CPOE) systems, given that prescribing and administration errors are a significant cause of morbidity and mortality. Of note, there is still much controversy surrounding the contention that poorly designed, implemented or used CDSS may actually lead to harm.

CONCLUSIONS

CDSS research remains an active and evolving area of research, as CDSS penetrate more widely beyond their traditional domain into consumer decision support, and as decisions become more complex, for example by involving sequence level genetic data.

The Centre for Health Informatics at the University of New South Wales - a Clinical Informatics Research Centre

Building a sustainable health system in the 21st Century will require the reinvention of much of the present day system, and the intelligent use of information and communication technologies (ICT) to deliver high quality, safe, efficient and affordable health care. The Centre for Health Informatics (CHI) is Australia’s largest academic research group in this emerging discipline.

Our research is underpinned by a planning process, based on different future scenarios for the health system, which helps us identify longer-term problems needing a sustained research effort. A research competency matrix is used to ensure that the Centre has the requisite core capabilities in the research methods and tools needed to pursue our research program.

The Centre’s work is internationally recognized for its contributions in the development of intelligent search systems to support evidence-based healthcare, developing evaluation methodologies for ICT, and in understanding how communication shapes the safety and quality of health care delivery. Centre researchers also are working on safety models and standards for ICT in healthcare, mining complex gene micro array, medical literature and medical record data, building health system simulation methods to model the impact of health policy changes, and developing novel computational methods to automate the diagnosis of 3-D medical images.

Any individual research group like CHI must necessarily focus on a few areas to allow it to develop sufficient research capacity to make novel and internationally significant contributions. As CHI approaches the end of its first decade, it is becoming clear that developing capacity becomes increasingly challenging as the research territory changes under our feet, and that the Centre will continue to evolve and shift its focus in the years to come.

AFL and FRL: abstraction and representation for field interchange

The holy grail of biomedical modelling is an integrated model of the entire human body. To this end, research groups around the world need to interchange experimental data, models and model results. A good interchange will have an efficient representation for storage and sharing and will have tools for modelling, data verification, authoring, data conversions and so on. A field is a spatially varying properly. In this paper we present the abstract field layer (AFL) and the field representation language (FRL). The AFL provides the field abstraction together with a set of common field operations. The FRL provides an efficient means for field representation and storage. We show how fields can be used to interchange information between modelling systems and between modelling and visualisation systems. We are currently developing a software system that composes multiple single cell solvers to create a tissue solver.

A three-dimensional fractal model of tumour vasculature

We constructed a three-dimensional fractal model of the vascular network in a tumour periphery. We model the highly disorganised structure of the neoplastic vasculature by using a high degree of variation in segment properties such as length, diameter and branching angle. The overall appearance of the vascular tree is subjectively similar to that of the disorganised vascular network which encapsulates tumours. The fractal dimension of the model is within the range of clinically measured values.

The field representation language

The complexity of quantitative biomedical models, and the rate at which they are published, is increasing to a point where managing the information has become all but impossible without automation. International efforts are underway to standardise representation languages for a number of mathematical entities that represent a wide variety of physiological systems. This paper presents the Field Representation Language (FRL), a portable representation of values that change over space and/or time. FRL is an extensible mark-up language (XML) derivative with support for large numeric data sets in Hierarchical Data Format version 5 (HDF5). Components of FRL can be reused through unified resource identifiers (URI) that point to external resources such as custom basis functions, boundary geometries and numerical data. To demonstrate the use of FRL as an interchange we present three models that study hyperthermia cancer treatment: a fractal model of liver tumour microvasculature; a probabilistic model simulating the deposition of magnetic microspheres throughout it; and a finite element model of hyperthermic treatment. The microsphere distribution field was used to compute the heat generation rate field around the tumour. We used FRL to convey results from the microsphere simulation to the treatment model. FRL facilitated the conversion of the coordinate systems and approximated the integral over regions of the microsphere deposition field.

The Centre for Health Informatics at the University of New South Wales--a clinical informatics research centre

OBJECTIVES

Building a sustainable health system in the 21st Century will require the reinvention of much of the present day system, and the intelligent use of information and communication technologies (ICT) to deliver high quality, safe, efficient and affordable health care. The Centre for Health Informatics (CHI) is Australia's largest academic research group in this emerging discipline.

METHODS

Our research is underpinned by a planning process, based on different future scenarios for the health system, which helps us identify longer-term problems needing a sustained research effort. A research competency matrix is used to ensure that the Centre has the requisite core capabilities in the research methods and tools needed to pursue our research program.

RESULTS

The Centre's work is internationally recognized for its contributions in the development of intelligent search systems to support evidence-based healthcare, developing evaluation methodologies for ICT, and in understanding how communication shapes the safety and quality of health care delivery. Centre researchers also are working on safety models and standards for ICT in healthcare, mining complex gene micro array, medical literature and medical record data, building health system simulation methods to model the impact of health policy changes, and developing novel computational methods to automate the diagnosis of 3-D medical images.

CONCLUSIONS

Any individual research group like CHI must necessarily focus on a few areas to allow it to develop sufficient research capacity to make novel and internationally significant contributions. As CHI approaches the end of its first decade, it is becoming clear that developing capacity becomes increasingly challenging as the research territory changes under our feet, and that the Centre will continue to evolve and shift its focus in the years to come.

Modelling heating of liver tumours with heterogeneous magnetic microsphere deposition

Ferromagnetic embolization hyperthermia (FEH) is a novel treatment for liver cancer. Magnetic microspheres are injected into the hepatic artery and cluster in the periphery of tumours and are heated with externally applied magnetic fields. In order to more accurately simulate FEH, we modelled a three-dimensional heterogeneous distribution of heat sources. We constructed a fractal model of the vasculature in the periphery of a tumour. We used this model to compute the spatial distribution of the microspheres that lodge in capillaries. We used the distribution model as input to a finite-element heat transfer model of the FEH treatment. The overall appearance of the vascular tree is subjectively similar to that of the disorganized vascular network which encapsulates tumours. The microspheres are distributed in the tumour periphery in similar patterns to experimental observations. We expect the vasculature and microsphere deposition models to also be of interest to researchers of any targeted cancer therapies such as localized intra-arterial chemotherapy and selective internal radiotherapy. Our results show that heterogeneous microsphere distributions give significantly different results to those for a homogeneous model and thus are preferable when accurate results are required.