Evaluating gold standard corpora against gene/protein tagging solutions and lexical resources

Motivation

The identification of protein and gene names (PGNs) from the scientific literature requires semantic resources: Terminological and lexical resources deliver the term candidates into PGN tagging solutions and the gold standard corpora (GSC) train them to identify term parameters and contextual features. Ideally all three resources, i.e. corpora, lexica and taggers, cover the same domain knowledge, and thus support identification of the same types of PGNs and cover all of them. Unfortunately, none of the three serves as a predominant standard and for this reason it is worth exploring, how these three resources comply with each other. We systematically compare different PGN taggers against publicly available corpora and analyze the impact of the included lexical resource in their performance. In particular, we determine the performance gains through false positive filtering, which contributes to the disambiguation of identified PGNs.

Results

In general, machine learning approaches (ML-Tag) for PGN tagging show higher F1-measure performance against the BioCreative-II and Jnlpba GSCs (exact matching), whereas the lexicon based approaches (LexTag) in combination with disambiguation methods show better results on FsuPrge and PennBio. The ML-Tag solutions balance precision and recall, whereas the LexTag solutions have different precision and recall profiles at the same F1-measure across all corpora. Higher recall is achieved with larger lexical resources, which also introduce more noise (false positive results). The ML-Tag solutions certainly perform best, if the test corpus is from the same GSC as the training corpus. As expected, the false negative errors characterize the test corpora and – on the other hand – the profiles of the false positive mistakes characterize the tagging solutions. Lex-Tag solutions that are based on a large terminological resource in combination with false positive filtering produce better results, which, in addition, provide concept identifiers from a knowledge source in contrast to ML-Tag solutions.

Conclusion

The standard ML-Tag solutions achieve high performance, but not across all corpora, and thus should be trained using several different corpora to reduce possible biases. The LexTag solutions have different profiles for their precision and recall performance, but with similar F1-measure. This result is surprising and suggests that they cover a portion of the most common naming standards, but cope differently with the term variability across the corpora. The false positive filtering applied to LexTag solutions does improve the results by increasing their precision without compromising significantly their recall. The harmonisation of the annotation schemes in combination with standardized lexical resources in the tagging solutions will enable their comparability and will pave the way for a shared standard.

Evaluation and cross-comparison of lexical entities of biological interest (LexEBI)

MOTIVATION

Biomedical entities, their identifiers and names, are essential in the representation of biomedical facts and knowledge. In the same way, the complete set of biomedical and chemical terms, i.e. the biomedical "term space" (the "Lexeome"), forms a key resource to achieve the full integration of the scientific literature with biomedical data resources: any identified named entity can immediately be normalized to the correct database entry. This goal does not only require that we are aware of all existing terms, but would also profit from knowing all their senses and their semantic interpretation (ambiguities, nestedness).

RESULT

This study compiles a resource for lexical terms of biomedical interest in a standard format (called "LexEBI"), determines the overall number of terms, their reuse in different resources and the nestedness of terms. LexEBI comprises references for protein and gene entries and their term variants and chemical entities amongst other terms. In addition, disease terms have been identified from Medline and PubmedCentral and added to LexEBI. Our analysis demonstrates that the baseforms of terms from the different semantic types show only little polysemous use. Nonetheless, the term variants of protein and gene names (PGNs) frequently contain species mentions, which should have been avoided according to protein annotation guidelines. Furthermore, the protein and gene entities as well as the chemical entities, both do comprise enzymes leading to hierarchical polysemy, and a large portion of PGNs make reference to a chemical entity. Altogether, according to our analysis based on the Medline distribution, 401,869 unique PGNs in the documents contain a reference to 25,022 chemical entities, 3,125 disease terms or 1,576 species mentions.

CONCLUSION

LexEBI delivers the complete biomedical and chemical Lexeome in a standardized representation (http://www.ebi.ac.uk/Rebholz-srv/LexEBI/). The resource provides the disease terms as open source content, and fully interlinks terms across resources.

Monitoring named entity recognition: the League Table

Background

Named entity recognition (NER) is an essential step in automatic text processing pipelines. A number of solutions have been presented and evaluated against gold standard corpora (GSC). The benchmarking against GSCs is crucial, but left to the individual researcher. Herewith we present a League Table web site, which benchmarks NER solutions against selected public GSCs, maintains a ranked list and archives the annotated corpus for future comparisons.

Results

The web site enables access to the different GSCs in a standardized format (IeXML). Upon submission of the annotated corpus the user has to describe the specification of the used solution and then uploads the annotated corpus for evaluation. The performance of the system is measured against one or more GSCs and the results are then added to the web site (“League Table”). It displays currently the results from publicly available NER solutions from the Whatizit infrastructure for future comparisons.

Conclusion

The League Table enables the evaluation of NER solutions in a standardized infrastructure and monitors the results long-term. For access please go to http://wwwdev.ebi.ac.uk/Rebholz-srv/calbc/assessmentGSC/. Contact: rebholz@ifi.uzh.ch.

Harmonization of gene/protein annotations: towards a gold standard MEDLINE

MOTIVATION

The recognition of named entities (NER) is an elementary task in biomedical text mining. A number of NER solutions have been proposed in recent years, taking advantage of available annotated corpora, terminological resources and machine-learning techniques. Currently, the best performing solutions combine the outputs from selected annotation solutions measured against a single corpus. However, little effort has been spent on a systematic analysis of methods harmonizing the annotation results and measuring against a combination of Gold Standard Corpora (GSCs).

RESULTS

We present Totum, a machine learning solution that harmonizes gene/protein annotations provided by heterogeneous NER solutions. It has been optimized and measured against a combination of manually curated GSCs. The performed experiments show that our approach improves the F-measure of state-of-the-art solutions by up to 10% (achieving ≈70%) in exact alignment and 22% (achieving ≈82%) in nested alignment. We demonstrate that our solution delivers reliable annotation results across the GSCs and it is an important contribution towards a homogeneous annotation of MEDLINE abstracts.

AVAILABILITY AND IMPLEMENTATION

Totum is implemented in Java and its resources are available at http://bioinformatics.ua.pt/totum

Assessment of NER solutions against the first and second CALBC Silver Standard Corpus

Background

Competitions in text mining have been used to measure the performance of automatic text processing solutions against a manually annotated gold standard corpus (GSC). The preparation of the GSC is time-consuming and costly and the final corpus consists at the most of a few thousand documents annotated with a limited set of semantic groups. To overcome these shortcomings, the CALBC project partners (PPs) have produced a large-scale annotated biomedical corpus with four different semantic groups through the harmonisation of annotations from automatic text mining solutions, the first version of the Silver Standard Corpus (SSC-I). The four semantic groups are chemical entities and drugs (CHED), genes and proteins (PRGE), diseases and disorders (DISO) and species (SPE). This corpus has been used for the First CALBC Challenge asking the participants to annotate the corpus with their text processing solutions.

Results

All four PPs from the CALBC project and in addition, 12 challenge participants (CPs) contributed annotated data sets for an evaluation against the SSC-I. CPs could ignore the training data and deliver the annotations from their genuine annotation system, or could train a machine-learning approach on the provided pre-annotated data. In general, the performances of the annotation solutions were lower for entities from the categories CHED and PRGE in comparison to the identification of entities categorized as DISO and SPE. The best performance over all semantic groups were achieved from two annotation solutions that have been trained on the SSC-I.

The data sets from participants were used to generate the harmonised Silver Standard Corpus II (SSC-II), if the participant did not make use of the annotated data set from the SSC-I for training purposes. The performances of the participants’ solutions were again measured against the SSC-II. The performances of the annotation solutions showed again better results for DISO and SPE in comparison to CHED and PRGE.

Conclusions

The SSC-I delivers a large set of annotations (1,121,705) for a large number of documents (100,000 Medline abstracts). The annotations cover four different semantic groups and are sufficiently homogeneous to be reproduced with a trained classifier leading to an average F-measure of 85%. Benchmarking the annotation solutions against the SSC-II leads to better performance for the CPs’ annotation solutions in comparison to the SSC-I.

UKPMC: a full text article resource for the life sciences

UK PubMed Central (UKPMC) is a full-text article database that extends the functionality of the original PubMed Central (PMC) repository. The UKPMC project was launched as the first ‘mirror’ site to PMC, which in analogy to the International Nucleotide Sequence Database Collaboration, aims to provide international preservation of the open and free-access biomedical literature. UKPMC (http://ukpmc.ac.uk) has undergone considerable development since its inception in 2007 and now includes both a UKPMC and PubMed search, as well as access to other records such as Agricola, Patents and recent biomedical theses. UKPMC also differs from PubMed/PMC in that the full text and abstract information can be searched in an integrated manner from one input box. Furthermore, UKPMC contains ‘Cited By’ information as an alternative way to navigate the literature and has incorporated text-mining approaches to semantically enrich content and integrate it with related database resources. Finally, UKPMC also offers added-value services (UKPMC+) that enable grantees to deposit manuscripts, link papers to grants, publish online portfolios and view citation information on their papers. Here we describe UKPMC and clarify the relationship between PMC and UKPMC, providing historical context and future directions, 10 years on from when PMC was first launched.

Natural language processing in aid of FlyBase curators

Background

Despite increasing interest in applying Natural Language Processing (NLP) to biomedical text, whether this technology can facilitate tasks such as database curation remains unclear.

Results

PaperBrowser is the first NLP-powered interface that was developed under a user-centered approach to improve the way in which FlyBase curators navigate an article. In this paper, we first discuss how observing curators at work informed the design and evaluation of PaperBrowser. Then, we present how we appraise PaperBrowser's navigational functionalities in a user-based study using a text highlighting task and evaluation criteria of Human-Computer Interaction. Our results show that PaperBrowser reduces the amount of interactions between two highlighting events and therefore improves navigational efficiency by about 58% compared to the navigational mechanism that was previously available to the curators. Moreover, PaperBrowser is shown to provide curators with enhanced navigational utility by over 74% irrespective of the different ways in which they highlight text in the article.

Conclusion

We show that state-of-the-art performance in certain NLP tasks such as Named Entity Recognition and Anaphora Resolution can be combined with the navigational functionalities of PaperBrowser to support curation quite successfully.

Integrating natural language processing with FlyBase curation

Applying Natural Language Processing techniques to biomedical text as a potential aid to curation has become the focus of intensive research. However, developing integrated systems which address the curators' real-world needs has been studied less rigorously. This paper addresses this question and presents generic tools developed to assist FlyBase curators. We discuss how they have been integrated into the curation workflow and present initial evidence about their effectiveness.

Bootstrapping the recognition and anaphoric linking of named entities in Drosophila articles

This paper demonstrates how Drosophila gene name recognition and anaphoric linking of gene names and their products can be achieved using existing information in FlyBase and the Sequence Ontology. Extending an extant approach to gene name recognition we achieved a F-score of 0.8559, and we report a preliminary experiment using a baseline anaphora resolution algorithm. We also present guidelines for annotation of gene mentions in texts and outline how the resulting system is used to aid FlyBase curation.

Levels of substance P and IgE decapeptide in nasal polyp fluid and matching sera: a preliminary study

Sixteen unselected patients with nasal polyps had the levels of substance P and IgE decapeptide measured by ELISA in the oedema fluids and their matched sera. All 16 samples had low levels of substance P in their sera and had high level of substance P in eight samples of nasal polyp oedema. There was a considerable variation in the values of IgE decapeptide found in the sera but 14 polyp oedema fluids had high levels of IgE decapeptide. This study supports the idea that there is a linkage between the cellular and neurovascular responses. High levels of IgE decapeptide suggest that mast cell reactions occur in the majority of cases and that IgE may be implicated in the process of mast cell degranulation.

Aggregation of the Fc epsilon RI in mast cells induces the synthesis of Fos-interacting protein and increases its DNA binding-activity: the dependence on protein kinase C-beta

The ability of c-Fos to dimerize with various proteins creates transcription complexes which can exert their regulatory function on a variety of genes. One of the transcription factors that binds to c-Fos is the newly discovered Fos-interacting protein (FIP). In this report we present evidence for the regulation of the synthesis of FIP by a physiological stimulus. We found that the aggregation of the mast cell high affinity receptor for IgE (Fc epsilon RI) induced the synthesis of FIP and increased its DNA binding activity. Moreover, down-regulation of the isoenzyme protein kinase C-beta (PKC-beta) by a specific antisense phosphorothioate oligonucleotide resulted in profound inhibition of FIP-Fos DNA binding activity. Thus, aggregation of the Fc epsilon RI on mast cells elicits a PKC-beta dependent signaling pathway which regulates FIP-Fos DNA binding activity.

The impact of missile warfare on self-reported sleep quality. Part 1

During the 1991 Gulf War, we investigated the effect of missile attacks through two telephone surveys of a large sample of an urban population that evaluated self-reported sleep quality, stress, fear, depressed mood, fatigue and power of concentration. We surveyed 1,045 people during the Gulf War itself, and we interviewed them again (excluding the chronic insomniacs) 30 days after the war. During the war, 51% of the subjects claimed to be suffering from disturbed sleep. Whereas 13% of the survey population had been chronic insomniacs before the war, 38% developed insomnia during the war. The war provoked reported stress (67.5% of subjects), depressed mood (50.9%), difficulties in concentration (39.7%) and increased fatigue (25%). Four weeks after it ended, 19% of the previously normal subjects were still suffering from insomnia; 5% of the cases of insomnia were developed postbellum. Stress, depressed mood and impaired concentration were found to correlate significantly with subjectively evaluated insomnia. We concluded that modern missile warfare may induce long-lasting insomnia in one-third of the population under threat. A small percentage may develop insomnia postbellum. The risk of developing long-lasting insomnia is higher in those who reported experiencing prolonged stress and depressed moods.

Molecules, management and medical outcomes: an international review

The first major joint conference between the Royal College of Physicians of London and the American College of Physicians was held at the Royal College of Physicians on 7-8 June 1993. The large enthusiastic audience from the UK and the USA demonstrated the cordiality which exists between the two colleges. The objective of the conference was to further an exchange of ideas about the influence of science and technology upon current and future medical practice. Four major areas were chosen for review: diabetes, viral hepatitis, cerebrovascular disease and asthma. Presentations within each area were devoted first to scientific principles, secondly to aspects of clinical management, and finally to issues of clinical outcome.

Regulation of AP-1 expression and activity in antigen-stimulated mast cells: the role played by protein kinase C and the possible involvement of Fos interacting protein

We have recently observed that protein kinase C (PKC) was involved in the regulation of the accumulation of mRNAs of the AP-1 components in cultured Abelson-transformed murine fetal-liver-derived mast cells stimulated by exocytotic stimuli. Here we analyzed the probable regulatory effect of PKC on the synthesis and DNA-binding activity of AP-1 complexes in immunologic stimulated mast cells. In this study we used the interleukin-3--dependent murine fetal-liver--derived mast cells that were not transformed by the Abelson oncogene. Study of PKC-depleted cells showed PKC dependency of c-fos mRNA accumulation and protein expression in IgE-Ag stimulated cells. In contrast, the c-jun mRNA accumulation was unaffected by PKC depletion, whereas its protein expression was dependent on this enzymatic activity. This suggests the involvement of PKC in the regulation of translation of c-Jun, a level of c-Jun regulation that was not previously described. The amount of AP-1 DNA-bound complex was also lowered in PKC-depleted cells. Therefore, PKC plays an important regulatory role in different stages of the signal transduction pathway because of IgE-Ag stimulation. Surprisingly, we have observed that although the amount of total synthesized c-Fos began to increase 15 minutes after immunologic stimulation, the amount of c-Fos associated with Juns did not increase, even after 45 minutes. This association was not affected by PKC. Using a Fos-interacting protein (FIP)-cDNA probe, an expression of 2.9 kb mRNA was detected in these cells. Furthermore, immunologic stimulation caused an increase in the amount of a Fos-containing protein complex that bound to an FIP-binding DNA oligonucleotide. Therefore, we propose that this protein complex that contains most of the immunologically induced c-Fos has an important role in IgE-Ag-stimulated signal transduction.

Serum modulates mast cell responses to IgE antigen stimulation

Serum induces the expression of the fos and jun gene families, which encode the transcription factor AP-1. Since we previously found that activation of mast cells by IgE-antigen (Ag) induces the mRNA accumulation of c-fos, c-jun, junB and junD proto-oncogenes, we were prompted to investigate whether serum could affect such accumulation in these cells. In addition, we investigated whether serum could modulate inhibition of DNA synthesis in immunologically stimulated mast cells. Mast cells, which were cultured in the presence of fetal calf serum (FCS), were characterized by a high proliferation rate and high accumulation of the mRNA of c-fos, junB and junD proto-oncogenes. After sustained FCS deprivation both DNA synthesis and the level of c-fos mRNA were significantly decreased, as expected, whereas the level of c-jun, junB and junD mRNA were not affected. As opposed to mast cells which were cultured in the presence of FCS, immunological stimulation of FCS-deprived cells resulted in DNA synthesis inhibition and an increase in c-fos expression. The results also show that the level of c-fos mRNA was increased by either IgE-Ag or FCS up to a similar level, while these two triggers could not act synergistically to enhance this expression further. Thus, changes in DNA synthesis, induced by FCS, block the ability of the immunological challenge to inhibit mast cell growth and to enhance c-fos mRNA accumulation.

Emotional experiences during the three trimesters of pregnancy

This study examined the emotional changes that occur during the trimesters of pregnancy. Two hundred eighty-two women were asked, one day after giving birth, to indicate at what frequency they had experienced various symptoms during each trimester of pregnancy and to fill out the Repression-Sensitization scale (Byrne, Barry, & Nelson, 1963). Results showed that while women's feelings during the first trimester are characterized by symptoms related to physiological changes (e.g., nausea, vomiting, dizziness), during the last trimester anxiety and emotional distress become the most significant symptoms. The level at which these symptoms were experienced was affected by the subject's socioeconomic level, number of previous births (primaparae or multiparae), and her personality type (repressor or sensitizer).

The association and predictive value of the complex immunoglobulin A-alpha 1-antitrypsin in the development of erosions in early rheumatoid arthritis

Immunoglobulin A-alpha 1 antitrypsin complex (IgA-AT), its constituent components and nine other clinical or laboratory variables were measured in thirty-three patients with early, non-erosive rheumatoid arthritis (RA) in order to assess their value in predicting the subsequent development of erosions. After 12 months, eighteen patients had developed erosions. Comparison of variables measured at outset between the group of patients subsequently developing erosions and those not, showed only the complex IgA-AT level to be significantly different, the mean being higher in the erosive group. In the subgroup of patients with high IgA-AT levels (greater than 3.0 arbitary units) all developed erosions. The possible therapeutic implications of these findings are discussed.

Development and evaluation of a rapid, semi-automatic micro-method for CH50 estimation using a computer program

Established methods for the estimation of serum complement are often unsatisfactory. Problems include complex mathematical and/or technical manipulations, lack of objectivity, and poor sensitivity. Here we present an assay that is rapid, sensitive, quantitative, simple and semi-automatic by using an 'ELISA' reader to estimate released haemoglobin. It compares very favourable with a more manual, old established method. We have used this new method to establish a normal range, investigate serum storage conditions, and demonstrate that the sensitised sheep red blood cells are suitable targets after overnight storage at 4 degrees C. Furthermore we confirm that serum from patients with systemic lupus erythematosus or Sjögren's syndrome frequently has reduced levels of CH50. Patients with rheumatoid arthritis, scleroderma, Bechet's disease or arteritis have a mean CH50 within the normal range.

Affiliation patterns among cancer patients

Affiliation preferences of 150 cancer out-patients were assessed in the waiting-room and day-to-day life. Results indicated the vast majority avoided fellow patients, preferring to be alone or with healthy people. Similarly, patients tended to avoid the topic of illness in their social contacts. Moreover, those desiring to be with the healthy displayed the highest levels of negative emotion. Finally, avoidance of fellow patients was higher among males, the elderly, and the more highly educated. Overall, the findings were inconsistent with the thesis that those under stress seek out others in similar straits, as suggested by Schachter's emotional comparison theory or Will's downward comparison theory. Rather, they support Rofé's utility theory of stress and affiliation.

Internal versus external references with respect to perception of stimuli on the body surface

Previous studies on the determinants of locus of perception of stimuli on the body have suggested that the position of the stimulus has a significant effect whether a subject perceives a tactile pattern as seen from inside or outside the body. However, it is possible that previous investigators confounded stimulus location--dorsal or frontal--and experimenter's position--behind or in front of the subject. Using 42 male subjects in a 2 X 2 design, the effects of experimenter's position and stimulus location were studied by a new technique for inferring locus of perception. Experimenter's position, rather than stimulus location, affects subjects' locus of perception. Perception of stimuli on the body involves three independent factors, the demand characteristics of the experiment, the manner in which an individual perceives the boundaries of his own body, and an individual's ability to adopt the experimenter's perceptual standpoint.

The Goldstein-Scheerer Cube Test: a developmental-regressive study

The Goldstein-Scheerer Cube Test was analyzed within a developmental frame of reference because Goldstein and Scheerer's criteria of pathological thought processes were found to be parallel to stages of development in concept formation. In addition to accuracy of the reproduction, the shape of the design frame and any rotations also were studied. Hypotheses were that the younger the subjects, the more global and diffuse their reproduction, while with increasing age the designs would be more accurate. A developmental gradient was suggested on a priori theoretical considerations. Thus, a gradient that consisted of three stages of perceptual matching, superceded by five stages of abstract analysis of the design, was envisaged. With regard to the frame and position of the reproduction, the breaking up of the square frame was considered most regressive. Among the rotations, those of 90%-180% were seen as more regressive than those of 45%; distinctions were made according to the causes of the rotations. The test was administered to samples of children aged 5, 7, 9, and 11 years, respectively (N = 171). The hypotheses generally were supported. The stages of the gradient of the configuration of the pattern showed very high scalability. The ranking among the shape of the frame of the copies and the various rotations of the copies relative to the design were supported. The 5-year-olds were significantly different from the three older age groups, as predicted. The results indicate a regressive hierarchy among faulty reactions to the Cube Test.

Patient affiliation in major illness

Affiliation preferences among 100 Israeli out-patients with major illness were assessed in waiting room interviews. Patients preferred healthy others over being with fellow sufferers or alone, and avoided illness as a discussion topic. Secondary preference for being alone rather than with co-patients rose slightly with reported anxiety. Findings contradict the emotional comparison hypothesis that misery seeks out co-sufferers.

Measurement of IgA-alpha 1 anti-trypsin (alpha 1 AT) complex in the sera of patients with IgA myelomatosis

The level of IgA-alpha 1 anti-trypsin (alpha 1 AT) complex in a relatively large number of IgA myeloma sera has been determined, and compared with their content of polymerised forms of IgA. The level of the complex was the same in sera containing only monomeric IgA, some polymer and more than 50% polymer (as determined by SDS-PAGE). There was, however, a highly significant inverse correlation between the amount of IgA-alpha 1 AT complex in the myeloma sera and their content of 10S dimer (as determined by analytical ultracentrifugation). High levels of IgA-alpha 1 AT complex were also found in the small number of myeloma sera examined which contained paraprotein of the minor allotypic form of (Am2+) of the IgA2 sub-class, indicating that the lack of disulphide bonds between the heavy and light chains of this isotype has no influence on its ability to complex with alpha 1 AT.

Affiliation before and after child delivery as a function of repression-sensitization

A group of 177 women were asked, before and after child delivery, to state their preference for being alone or with other women. They were also asked to rank their level of anxiety. After delivery they filled out the Repression-Sensitization Scale of Byrne, Barry & Nelson (1963). Results show sensitizers to be more anxious than repressors before but not after delivery. Before delivery repressors preferred to be alone, while sensitizers wanted to stay with others. After delivery both groups show a significant preference for being with others. There is no significant correlation between anxiety and affiliation.

REM and dreaming

4 subjects were awakened after long REM periods and 12 other subjects after short REM and after NREMs. The mean numbers of words, dream episodes, and spontaneous remarks about being interrupted in the middle of a dream were significantly different in the reports of the three groups. It is suggested that the present view of the relationship between REM and dreaming is not conclusive.