Quality Assurance of UMLS Semantic Type Assignments Using SNOMED CT Hierarchies

An Automated Approach for Identifying Erroneous IS-A Relations in SNOMED CT

SNOMED CT is the most comprehensive clinical terminology employed worldwide and enhancing its accuracy is of utmost importance. In this work, we introduce an automated approach to identifying erroneous IS-A relations in SNOMED CT. We first extract linked concept-pairs from which we generate Term Difference Pairs (TDPs) that contain differences between the concepts. Given a TDP, if the reversed TDP also exists and the number of linked-pairs generating this TDP is less than those generating the reversed TDP, then we suggest the former linked-pairs as potentially erroneous IS-A relations. We applied this approach to the Clinical finding and Procedure subhierarchies of the 2022 March US Edition of SNOMED CT, and obtained 52 potentially erroneous IS-A relations and a candidate list of 48 linked-pairs. A domain expert confirmed 41 out of 52 (78.8%) are valid and identified 26 erroneous IS-A relations out of 48 linked-pairs demonstrating the effectiveness of the approach.

The Unified Medical Language System at 30 Years and How It Is Used and Published: Systematic Review and Content Analysis

BACKGROUND

The Unified Medical Language System (UMLS) has been a critical tool in biomedical and health informatics, and the year 2021 marks its 30th anniversary. The UMLS brings together many broadly used vocabularies and standards in the biomedical field to facilitate interoperability among different computer systems and applications.

OBJECTIVE

Despite its longevity, there is no comprehensive publication analysis of the use of the UMLS. Thus, this review and analysis is conducted to provide an overview of the UMLS and its use in English-language peer-reviewed publications, with the objective of providing a comprehensive understanding of how the UMLS has been used in English-language peer-reviewed publications over the last 30 years.

METHODS

PubMed, ACM Digital Library, and the Nursing & Allied Health Database were used to search for studies. The primary search strategy was as follows: UMLS was used as a Medical Subject Headings term or a keyword or appeared in the title or abstract. Only English-language publications were considered. The publications were screened first, then coded and categorized iteratively, following the grounded theory. The review process followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines.

RESULTS

A total of 943 publications were included in the final analysis. Moreover, 32 publications were categorized into 2 categories; hence the total number of publications before duplicates are removed is 975. After analysis and categorization of the publications, UMLS was found to be used in the following emerging themes or areas (the number of publications and their respective percentages are given in parentheses): natural language processing (230/975, 23.6%), information retrieval (125/975, 12.8%), terminology study (90/975, 9.2%), ontology and modeling (80/975, 8.2%), medical subdomains (76/975, 7.8%), other language studies (53/975, 5.4%), artificial intelligence tools and applications (46/975, 4.7%), patient care (35/975, 3.6%), data mining and knowledge discovery (25/975, 2.6%), medical education (20/975, 2.1%), degree-related theses (13/975, 1.3%), digital library (5/975, 0.5%), and the UMLS itself (150/975, 15.4%), as well as the UMLS for other purposes (27/975, 2.8%).

CONCLUSIONS

The UMLS has been used successfully in patient care, medical education, digital libraries, and software development, as originally planned, as well as in degree-related theses, the building of artificial intelligence tools, data mining and knowledge discovery, foundational work in methodology, and middle layers that may lead to advanced products. Natural language processing, the UMLS itself, and information retrieval are the 3 most common themes that emerged among the included publications. The results, although largely related to academia, demonstrate that UMLS achieves its intended uses successfully, in addition to achieving uses broadly beyond its original intentions.

A review of auditing techniques for the Unified Medical Language System

OBJECTIVE

The study sought to describe the literature related to the development of methods for auditing the Unified Medical Language System (UMLS), with particular attention to identifying errors and inconsistencies of attributes of the concepts in the UMLS Metathesaurus.

MATERIALS AND METHODS

We applied the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach by searching the MEDLINE database and Google Scholar for studies referencing the UMLS and any of several terms related to auditing, error detection, and quality assurance. A qualitative analysis and summarization of articles that met inclusion criteria were performed.

RESULTS

Eighty-three studies were reviewed in detail. We first categorized techniques based on various aspects including concepts, concept names, and synonymy (n = 37), semantic type assignments (n = 36), hierarchical relationships (n = 24), lateral relationships (n = 12), ontology enrichment (n = 8), and ontology alignment (n = 18). We also categorized the methods according to their level of automation (ie, automated systematic, automated heuristic, or manual) and the type of knowledge used (ie, intrinsic or extrinsic knowledge).

CONCLUSIONS

This study is a comprehensive review of the published methods for auditing the various conceptual aspects of the UMLS. Categorizing the auditing techniques according to the various aspects will enable the curators of the UMLS as well as researchers comprehensive easy access to this wealth of knowledge (eg, for auditing lateral relationships in the UMLS). We also reviewed ontology enrichment and alignment techniques due to their critical use of and impact on the UMLS.

Incidence, risk factors and re-exacerbation rate of severe asthma exacerbations in a multinational, multidatabase pediatric cohort study

BACKGROUND

There are sparse real-world data on severe asthma exacerbations (SAE) in children. This multinational cohort study assessed the incidence of and risk factors for SAE and the incidence of asthma-related rehospitalization in children with asthma.

METHODS

Asthma patients 5-17 years old with ≥1 year of follow-up were identified in six European electronic databases from the Netherlands, Italy, the UK, Denmark and Spain in 2008-2013. Asthma was defined as ≥1 asthma-specific disease code within 3 months of prescriptions/dispensing of asthma medication. Severe asthma was defined as high-dosed inhaled corticosteroids plus a second controller. SAE was defined by systemic corticosteroids, emergency department visit and/or hospitalization all for reason of asthma. Risk factors for SAE were estimated by Poisson regression analyses.

RESULTS

The cohort consisted of 212 060 paediatric asthma patients contributing to 678 625 patient-years (PY). SAE rates ranged between 17 and 198/1000 PY and were higher in severe asthma and highest in severe asthma patients with a history of exacerbations. Prior SAE (incidence rate ratio 3-45) and younger age increased the SAE risk in all countries, whereas obesity, atopy and GERD were a risk factor in some but not all countries. Rehospitalization rates were up to 79% within 1 year.

CONCLUSIONS

In a real-world setting, SAE rates were highest in children with severe asthma with a history of exacerbations. Many severe asthma patients were rehospitalized within 1 year. Asthma management focusing on prevention of SAE is important to reduce the burden of asthma.

Multinational cohort study of mortality in patients with asthma and severe asthma

BACKGROUND

Data on the risk of death following an asthma exacerbation are scarce. With this multinational cohort study, we assessed all-cause mortality rates, mortality rates following an exacerbation, and patient characteristics associated with all-cause mortality in asthma.

METHODS

Asthma patients aged ≥18 years and with ≥1 year of follow-up were identified in 5 European electronic databases from the Netherlands, Italy, UK, Denmark and Spain during the study period January 1, 2008-December 31, 2013. Patients with asthma-COPD overlap were excluded. Severe asthma was defined as use of high dose ICS + use of a second controller. Severe asthma exacerbations were defined as emergency department visits, hospitalizations or systemic corticosteroid use, all for reason of asthma.

RESULTS

The cohort consisted of 586,436 asthma patients of which 42,611 patients (7.3%) had severe asthma. The age and sex standardized all-cause mortality rates ranged between databases from 5.2 to 9.5/1000 person-years (PY) in asthma, and between 11.3 and 14.8/1000 PY in severe asthma. The all-cause mortality rate in the first week following a severe asthma exacerbation ranged between 14.1 and 59.9/1000 PY. Mortality rates remained high in the first month following a severe asthma exacerbation and decreased thereafter. Higher age, male gender, comorbidity, smoking, and previous severe asthma exacerbations were associated with mortality.

CONCLUSION

All-cause mortality following a severe exacerbation is high, especially in the first month following the event. Smoking cessation, comorbidity-management and asthma-treatment focusing on the prevention of exacerbations might reduce associated mortality.

Comparison of the Results of Manual and Automated Processes of Cross-Mapping Between Nursing Terms: Quantitative Study

Background

Cross-mapping establishes equivalence between terms from different terminology systems, which is useful for interoperability, updated terminological versions, and reuse of terms. Due to the number of terms to be mapped, this work can be extensive, tedious, and thorough, and it is susceptible to errors; this can be minimized by automated processes, which use computational tools.

Objective

The aim of this study was to compare the results of manual and automated term mapping processes.

Methods

In this descriptive, quantitative study, we used the results of two mapping processes as an empirical basis: manual, which used 2638 terms of nurses’ records from a university hospital in southern Brazil and the International Classification for Nursing Practice (ICNP); and automated, which used the same university hospital terms and the primitive terms of the ICNP through MappICNP, an algorithm based on rules of natural language processing. The two processes were compared via equality and exclusivity assessments of new terms of the automated process and of candidate terms.

Results

The automated process mapped 569/2638 (21.56%) of the source bank’s terms as identical, and the manual process mapped 650/2638 (24.63%) as identical. Regarding new terms, the automated process mapped 1031/2638 (39.08%) of the source bank’s terms as new, while the manual process mapped 1251 (47.42%). In particular, manual mapping identified 101/2638 (3.82%) terms as identical and 429 (16.26%) as new, whereas the automated process identified 20 (0.75%) terms as identical and 209 (7.92%) as new. Of the 209 terms mapped as new by the automated process, it was possible to establish an equivalence with ICNP terms in 48 (23.0%) cases. An analysis of the candidate terms offered by the automated process to the 429 new terms mapped exclusively by the manual process resulted in 100 (23.3%) candidates that had a semantic relationship with the source term.

Conclusions

The automated and manual processes map identical and new terms in similar ways and can be considered complementary. Direct identification of identical terms and the offering of candidate terms through the automated process facilitate and enhance the results of the mapping; confirmation of the precision of the automated mapping requires further analysis by researchers.

MfeCNN: Mixture Feature Embedding Convolutional Neural Network for Data Mapping

Data mapping plays an important role in data integration and exchanges among institutions and organizations with different data standards. However, traditional rule-based approaches and machine learning methods fail to achieve satisfactory results for the data mapping problem. In this paper, we propose a novel and sophisticated deep learning framework for data mapping called mixture feature embedding convolutional neural network (MfeCNN). The MfeCNN model converts the data mapping task to a multiple classification problem. In the model, we incorporated multimodal learning and multiview embedding into a CNN for mixture feature tensor generation and classification prediction. Multimodal features were extracted from various linguistic spaces with a medical natural language processing package. Then, powerful feature embeddings were learned by using the CNN. As many as 10 classes could be simultaneously classified by a softmax prediction layer based on multiview embedding. MfeCNN achieved the best results on unbalanced data (average F1 score, 82.4%) among the traditional state-of-the-art machine learning models and CNN without mixture feature embedding. Our model also outperformed a very deep CNN with 29 layers, which took free texts as inputs. The combination of mixture feature embedding and a deep neural network can achieve high accuracy for data mapping and multiple classification.

Validating UMLS Semantic Type Assignments Using SNOMED CT Semantic Tags

BACKGROUND

The UMLS assigns semantic types to all its integrated concepts. The semantic types are widely used in various natural language processing tasks in the biomedical domain, such as named entity recognition, semantic disambiguation, and semantic annotation. Due to the size of the UMLS, erroneous semantic type assignments are hard to detect. It is imperative to devise automated techniques to identify errors and inconsistencies in semantic type assignments.

OBJECTIVES

Designing a methodology to perform programmatic checks to detect semantic type assignment errors for UMLS concepts with one or more SNOMED CT terms and evaluating concepts in a selected set of SNOMED CT hierarchies to verify our hypothesis that UMLS semantic type assignment errors may exist in concepts residing in semantically inconsistent groups.

METHODS

Our methodology is a four-stage process. 1) partitioning concepts in a SNOMED CT hierarchy into semantically uniform groups based on their assigned semantic tags; 2) partitioning concepts in each group from 1) into the disjoint sub-groups based on their semantic type assignments; 3) mapping all SNOMED CT semantic tags into one or more semantic types in the UMLS; 4) identifying semantically inconsistent groups that have inconsistent assignments between semantic tags and semantic types according to the mapping from 3) and providing concepts in such groups to the domain experts for reviewing.

RESULTS

We applied our method on the UMLS 2013AA release. Concepts of the semantically inconsistent groups in the PHYSICAL FORCE and RECORD ARTIFACT hierarchies have error rates 33% and 62.5% respectively, which are greatly larger than error rates 0.6% and 1% in semantically consistent groups of the two hierarchies.

CONCLUSION

Concepts in semantically in - consistent groups are more likely to contain semantic type assignment errors. Our methodology can make auditing more efficient by limiting auditing resources on concepts of semantically inconsistent groups.

Assessing the practice of biomedical ontology evaluation: Gaps and opportunities

With the proliferation of heterogeneous health care data in the last three decades, biomedical ontologies and controlled biomedical terminologies play a more and more important role in knowledge representation and management, data integration, natural language processing, as well as decision support for health information systems and biomedical research. Biomedical ontologies and controlled terminologies are intended to assure interoperability. Nevertheless, the quality of biomedical ontologies has hindered their applicability and subsequent adoption in real-world applications. Ontology evaluation is an integral part of ontology development and maintenance. In the biomedicine domain, ontology evaluation is often conducted by third parties as a quality assurance (or auditing) effort that focuses on identifying modeling errors and inconsistencies. In this work, we first organized four categorical schemes of ontology evaluation methods in the existing literature to create an integrated taxonomy. Further, to understand the ontology evaluation practice in the biomedicine domain, we reviewed a sample of 200 ontologies from the National Center for Biomedical Ontology (NCBO) BioPortal-the largest repository for biomedical ontologies-and observed that only 15 of these ontologies have documented evaluation in their corresponding inception papers. We then surveyed the recent quality assurance approaches for biomedical ontologies and their use. We also mapped these quality assurance approaches to the ontology evaluation criteria. It is our anticipation that ontology evaluation and quality assurance approaches will be more widely adopted in the development life cycle of biomedical ontologies.

A review of medical terminology standards and structured reporting

Much effort has been invested in standardizing medical terminology for representation of medical knowledge, storage in electronic medical records, retrieval, reuse for evidence-based decision making, and for efficient messaging between users. We only focus on those efforts related to the representation of clinical medical knowledge required for capturing diagnoses and findings from a wide range of general to specialty clinical perspectives (e.g., internists to pathologists). Standardized medical terminology and the usage of structured reporting have been shown to improve the usage of medical information in secondary activities, such as research, public health, and case studies. The impact of standardization and structured reporting is not limited to secondary activities; standardization has been shown to have a direct impact on patient healthcare.

Relating Complexity and Error Rates of Ontology Concepts. More Complex NCIt Concepts Have More Errors

OBJECTIVES

Ontologies are knowledge structures that lend support to many health-information systems. A study is carried out to assess the quality of ontological concepts based on a measure of their complexity. The results show a relation between complexity of concepts and error rates of concepts.

METHODS

A measure of lateral complexity defined as the number of exhibited role types is used to distinguish between more complex and simpler concepts. Using a framework called an area taxonomy, a kind of abstraction network that summarizes the structural organization of an ontology, concepts are divided into two groups along these lines. Various concepts from each group are then subjected to a two-phase QA analysis to uncover and verify errors and inconsistencies in their modeling. A hierarchy of the National Cancer Institute thesaurus (NCIt) is used as our test-bed. A hypothesis pertaining to the expected error rates of the complex and simple concepts is tested.

RESULTS

Our study was done on the NCIt's Biological Process hierarchy. Various errors, including missing roles, incorrect role targets, and incorrectly assigned roles, were discovered and verified in the two phases of our QA analysis. The overall findings confirmed our hypothesis by showing a statistically significant difference between the amounts of errors exhibited by more laterally complex concepts vis-à-vis simpler concepts.

CONCLUSIONS

QA is an essential part of any ontology's maintenance regimen. In this paper, we reported on the results of a QA study targeting two groups of ontology concepts distinguished by their level of complexity, defined in terms of the number of exhibited role types. The study was carried out on a major component of an important ontology, the NCIt. The findings suggest that more complex concepts tend to have a higher error rate than simpler concepts. These findings can be utilized to guide ongoing efforts in ontology QA.