Autonomous detection, grading, and reporting of postoperative complications using natural language processing

Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis

ABSTRACT

Venous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study's best performing model that evaluated for pulmonary embolism and/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95% CI, 0.881-0.962), specificity 0.984 (95% CI, 0.967-0.992), PPV 0.910 (95% CI, 0.865-0.941) and NPV 0.985 (95% CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.

Artificial intelligence in pediatric surgery

Artificial intelligence (AI) is rapidly changing the landscape of medicine and is already being utilized in conjunction with medical diagnostics and imaging analysis. We hereby explore AI applications in surgery and examine its relevance to pediatric surgery, covering its evolution, current state, and promising future. The various fields of AI are explored including machine learning and applications to predictive analytics and decision support in surgery, computer vision and image analysis in preoperative planning, image segmentation, surgical navigation, and finally, natural language processing assist in expediting clinical documentation, identification of clinical indications, quality improvement, outcome research, and other types of automated data extraction. The purpose of this review is to familiarize the pediatric surgical community with the rise of AI and highlight the ongoing advancements and challenges in its adoption, including data privacy, regulatory considerations, and the imperative for interdisciplinary collaboration. We hope this review serves as a comprehensive guide to AI's transformative influence on surgery, demonstrating its potential to enhance pediatric surgical patient outcomes, improve precision, and usher in a new era of surgical excellence.

Developing Validated Tools to Identify Pulmonary Embolism in Electronic Databases: Rationale and Design of the PE-EHR+ Study

BACKGROUND

 Contemporary pulmonary embolism (PE) research, in many cases, relies on data from electronic health records (EHRs) and administrative databases that use International Classification of Diseases (ICD) codes. Natural language processing (NLP) tools can be used for automated chart review and patient identification. However, there remains uncertainty with the validity of ICD-10 codes or NLP algorithms for patient identification.

METHODS

 The PE-EHR+ study has been designed to validate ICD-10 codes as Principal Discharge Diagnosis, or Secondary Discharge Diagnoses, as well as NLP tools set out in prior studies to identify patients with PE within EHRs. Manual chart review by two independent abstractors by predefined criteria will be the reference standard. Sensitivity, specificity, and positive and negative predictive values will be determined. We will assess the discriminatory function of code subgroups for intermediate- and high-risk PE. In addition, accuracy of NLP algorithms to identify PE from radiology reports will be assessed.

RESULTS

 A total of 1,734 patients from the Mass General Brigham health system have been identified. These include 578 with ICD-10 Principal Discharge Diagnosis codes for PE, 578 with codes in the secondary position, and 578 without PE codes during the index hospitalization. Patients within each group were selected randomly from the entire pool of patients at the Mass General Brigham health system. A smaller subset of patients will also be identified from the Yale-New Haven Health System. Data validation and analyses will be forthcoming.

CONCLUSIONS

 The PE-EHR+ study will help validate efficient tools for identification of patients with PE in EHRs, improving the reliability of efficient observational studies or randomized trials of patients with PE using electronic databases.

Classification of neurologic outcomes from medical notes using natural language processing

Neurologic disability level at hospital discharge is an important outcome in many clinical research studies. Outside of clinical trials, neurologic outcomes must typically be extracted by labor intensive manual review of clinical notes in the electronic health record (EHR). To overcome this challenge, we set out to develop a natural language processing (NLP) approach that automatically reads clinical notes to determine neurologic outcomes, to make it possible to conduct larger scale neurologic outcomes studies. We obtained 7314 notes from 3632 patients hospitalized at two large Boston hospitals between January 2012 and June 2020, including discharge summaries (3485), occupational therapy (1472) and physical therapy (2357) notes. Fourteen clinical experts reviewed notes to assign scores on the Glasgow Outcome Scale (GOS) with 4 classes, namely 'good recovery', 'moderate disability', 'severe disability', and 'death' and on the Modified Rankin Scale (mRS), with 7 classes, namely 'no symptoms', 'no significant disability', 'slight disability', 'moderate disability', 'moderately severe disability', 'severe disability', and 'death'. For 428 patients' notes, 2 experts scored the cases generating interrater reliability estimates for GOS and mRS. After preprocessing and extracting features from the notes, we trained a multiclass logistic regression model using LASSO regularization and 5-fold cross validation for hyperparameter tuning. The model performed well on the test set, achieving a micro average area under the receiver operating characteristic and F-score of 0.94 (95% CI 0.93-0.95) and 0.77 (0.75-0.80) for GOS, and 0.90 (0.89-0.91) and 0.59 (0.57-0.62) for mRS, respectively. Our work demonstrates that an NLP algorithm can accurately assign neurologic outcomes based on free text clinical notes. This algorithm increases the scale of research on neurological outcomes that is possible with EHR data.

Practice-Based Learning and Improvement: Improving Morbidity and Mortality Review Using Natural Language Processing

INTRODUCTION

Practice-Based Learning and Improvement, a core competency identified by the Accreditation Council for Graduate Medical Education, carries importance throughout a physician's career. Practice-Based Learning and Improvement is cultivated by a critical review of complications, yet methods to accurately identify complications are inadequate. Machine-learning algorithms show promise in improving identification of complications. We compare a manual-supplemented natural language processing (ms-NLP) methodology against a validated electronic morbidity and mortality (MM) database, the Morbidity and Mortality Adverse Event Reporting System (MARS) to understand the utility of NLP in MM review.

METHODS

The number and severity of complications were compared between MARS and ms-NLP of surgical hospitalization discharge summaries among three academic medical centers. Clavien-Dindo (CD) scores were assigned to cases with identified complications and classified into minor (CD I-II) or major (CD III-IV) harm.

RESULTS

Of 7774 admissions, 987 cases were identified to have 1659 complications by MARS and 1296 by ms-NLP. MARS identified 611 (62%) cases, whereas ms-NLP identified 670 (68%) cases. Less than one-third of cases (299, 30.3%) were detected by both methods. MARS identified a greater number of complications with major harm (457, 46.30%) than did ms-NLP (P < 0.0001).

CONCLUSIONS

Both a prospectively maintained MM database and ms-NLP review of discharge summaries fail to identify a significant proportion of postoperative complications and overlap 1/3 of the time. ms-NLP more frequently identifies cases with minor complications, whereas prospective voluntary reporting more frequently identifies major complications. The educational benefit of reporting and analysis of complication data may be supplemented by ms-NLP but not replaced by it at this time.

Development and validation of models for detection of postoperative infections using structured electronic health records data and machine learning

BACKGROUND

Postoperative infections constitute more than half of all postoperative complications. Surveillance of these complications is primarily done through manual chart review, which is time consuming, expensive, and typically only covers 10% to 15% of all operations. Automated surveillance would permit the timely evaluation of and reporting of all operations.

METHODS

The goal of this study was to develop and validate parsimonious, interpretable models for conducting surveillance of postoperative infections using structured electronic health records data. This was a retrospective study using 30,639 unique operations from 5 major hospitals between 2013 and 2019. Structured electronic health records data were linked to postoperative outcomes data from the American College of Surgeons National Surgical Quality Improvement Program. Predictors from the electronic health records included diagnoses, procedures, and medications. Infectious complications included surgical site infection, urinary tract infection, sepsis, and pneumonia within 30 days of surgery. The knockoff filter, a penalized regression technique that controls type I error, was applied for variable selection. Models were validated in a chronological held-out dataset.

RESULTS

Seven percent of patients experienced at least one type of postoperative infection. Models selected contained between 4 and 8 variables and achieved >0.91 area under the receiver operating characteristic curve, >81% specificity, >87% sensitivity, >99% negative predictive value, and 10% to 15% positive predictive value in a held-out test dataset.

CONCLUSION

Surveillance and reporting of postoperative infection rates can be implemented for all operations with high accuracy using electronic health records data and simple linear regression models.

Automated extraction of incidental adrenal nodules from electronic health records

BACKGROUND

Many adrenal incidentalomas do not undergo appropriate biochemical testing and complete imaging characterization to assess for hormone hypersecretion and malignancy. With the growing availability of clinical narratives in the electronic medical record, automated surveillance using advanced data analytic techniques may represent a promising method to improve management.

METHODS

A data provisioning process using a series of structured query language scripts was used to abstract all chest and abdominal computed tomography and magnetic resonance imaging reports from an academic health care system in 2018. The narratives and impressions were queried for key text relating to the identification of adrenal incidentalomas. Patients with a history of extra-adrenal malignancy undergoing staging or surveillance imaging were excluded. The prevalence and radiographic characteristics were analyzed. Patients with adrenal incidentalomas newly identified in 2018 were assessed for biochemical testing and nodule stability through August 2021.

RESULTS

Of 36,618 patients queried, 8,557 were excluded owing to a history of extra-adrenal malignancy. Data from 447 patients were flagged by the structured query language scripts and electronically abstracted. On internal validation, 307/447 (69%) patients were correctly identified as having adrenal nodules (1.1% overall prevalence). The median patient age was 67 years, and 56% were female. The median nodule size was 1.7 (IQR 1.3-2.5) cm, 9% were bilateral, and 63% were low density (unenhanced Hounsfield units <10). Adrenal carcinoma was identified in 10 patients. In 2018, 121 patients were diagnosed with a new adrenal incidentaloma. Of 32 (27%) patients who had follow-up imaging at a median of 1.9 years, 97% of nodules were stable in size. Biochemical testing was performed in 53 patients (44%), of which 31 (26%) had complete hormonal assessment; 14 (26%) were functional nodules: 7 aldosterone-secreting, 4 cortisol-secreting, and 3 pheochromocytoma.

CONCLUSION

Only one-fourth of patients received appropriate biochemical testing after incidental diagnosis of an adrenal nodule, and most nodules with indeterminate imaging characteristics did not undergo follow-up imaging. Advanced data analytic techniques on electronic imaging reports may aid in the clinical identification and improved management of patients with adrenal incidentalomas.

Development and validation of a model for surveillance of postoperative bleeding complications using structured electronic health records data

BACKGROUND

Postoperative bleeding complications surveillance is done primarily through manual chart review. The purpose of this study was to develop and validate a detection model for postoperative bleeding complications using structured electronic health records data.

METHODS

Patients who underwent operations at 1 of 5 hospitals within our local health system between 2013 and 2019 and whose complications were reported by the American College of Surgeons National Surgical Quality Improvement Program were included. Electronic health records data were linked to American College of Surgeons National Surgical Quality Improvement Program data using personal health identifiers. Electronic health records predictors included diagnosis codes mapped to PheCodes, procedure names, and medications within 30 days after surgery. We defined bleeding events as the transfusion of red blood cell components within 30 days after surgery. The knockoff filter and the lasso were used to develop a model in a training set of operations from January 2013 to March 2017. Performance of each model was tested in a held-out data set of patients who underwent operations from March 2017 to October 2019.

RESULTS

A total of 30,639 patients were included; 1,112 patients (3.6%) had a bleeding event. Eight predictor variables were selected by the knockoff filter. When applied to the test set, specificity was 94%, sensitivity was 94%, area under the curve was 0.97, and accuracy was 93%. Calibration was consistent in lower predicted risk patients, whereas the model slightly overpredicted risk in high-risk patients.

CONCLUSION

We created a parsimonious, accurate model for identifying patients with bleeding complications. This model can be used to augment manual chart review for surveillance and reporting of perioperative bleeding complications, enabling inclusion of all surgeries in quality improvement efforts.

Natural Language Processing for Identification of Venous Thromboembolism in a Clinical Decision Support System: Validation Study (Preprint)

BACKGROUND

It remains unknown whether capturing data from electronic health records (EHRs) using natural language processing (NLP) can improve venous thromboembolism (VTE) detection in different clinical settings.

OBJECTIVE

The aim of this study was to validate the NLP algorithm in a clinical decision support system for VTE risk assessment and integrated care (DeVTEcare) to identify VTEs from EHRs.

METHODS

All inpatients aged ≥18 years in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital from January 1 to December 31, 2021, were included as the validation cohort. The sensitivity, specificity, positive and negative likelihood ratios (LR+ and LR-, respectively), area under the receiver operating characteristic curve (AUC), and F1-scores along with their 95% CIs were used to analyze the performance of the NLP tool, with manual review of medical records as the reference standard for detecting deep vein thrombosis (DVT) and pulmonary embolism (PE). The primary end point was the performance of the NLP approach embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE among different hospital departments with different VTE risks. Subgroup analyses were performed among age, sex, and the study season.

RESULTS

Among 30,152 patients (median age 56 [IQR 41-67] years; 14,247/30,152, 47.3% females), the prevalence of VTE, PE, and DVT was 2.1% (626/30,152), 0.6% (177/30,152), and 1.8% (532/30,152), respectively. The sensitivity, specificity, LR+, LR-, AUC, and F1-score of NLP-facilitated VTE detection were 89.9% (95% CI 87.3%-92.2%), 99.8% (95% CI 99.8%-99.9%), 483 (95% CI 370-629), 0.10 (95% CI 0.08-0.13), 0.95 (95% CI 0.94-0.96), and 0.90 (95% CI 0.90-0.91), respectively. Among departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) were in the surgery department (all P<.001). Among low, intermediate, and high VTE risks in hospital departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively) and F1-score (0.97 vs 0.90 vs 0.90, respectively) as well as the lowest LR- (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC; all P<.001). Subgroup analysis of the age, sex, and season demonstrated consistently good performance of VTE detection with >87% sensitivity and specificity and >89% AUC and F1-score. The NLP algorithm performed better among patients aged ≤65 years than among those aged >65 years (F1-score 0.93 vs 0.89, respectively; P<.001).

CONCLUSIONS

The NLP algorithm in our DeVTEcare identified VTE well across different clinical settings, especially in patients in surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm can help to inform accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.

Racial differences in venous thromboembolism: A surveillance program in Durham County, North Carolina

Background

Venous thromboembolism (VTE) affects approximately 1-2 individuals per 1000 annually and is associated with an increased risk for pulmonary hypertension, postthrombotic syndrome, and recurrent VTE.

Objective

To determine risk factors, incidence, treatments, and outcomes of VTE through a 2-year surveillance program initiated in Durham County, North Carolina (population approximately 280,000 at time of study).

Patients/Methods

We performed a retrospective analysis of data actively collected from three hospitals in Durham County during the surveillance period.

Results

A total of 987 patients were diagnosed with VTE, for an annual rate of 1.76 per 1000 individuals. Hospital-associated VTE occurred in 167 hospitalized patients (16.9%) and 271 outpatients who were hospitalized within 90 days of diagnosis (27.5%). Annual incidence was 1.98 per 1000 Black individuals compared to 1.25 per 1000 White individuals (p < 0.0001), and Black individuals with VTE were younger than White individuals (p < 0.0001). Common risk factors included active cancer, prolonged immobility, and obesity, and approximately half were still taking anticoagulant therapy 1 year later. A total of 224 patients died by 1 year (28.5% of patients for whom outcomes could be confirmed), and Black patients were more likely to have recurrent VTE than White patients during the first 6 months following initial presentation (9.4% vs. 4.1%, p = 0.01).

Conclusions

Ongoing surveillance provides an effective strategy to identify patients with VTE and monitor treatment and outcomes. We demonstrated that hospital-associated VTE continues to be a major contributor to the burden of VTE and confirmed the higher incidence of VTE in Black compared to White individuals.

Applications of artificial intelligence in the diagnosis and prediction of erectile dysfunction: a narrative review

Despite the high prevalence of erectile dysfunction, patients are reluctant to seek medical advice, which leads to low diagnostic rates in clinical practice. Artificial intelligence has been widely applied in the diagnosis of many diseases and may alleviate the situation. However, the applications of artificial intelligence in erectile dysfunction have not been reviewed to date. Therefore, the assistance from artificial intelligence needs to be summarized. In this review, 418 publications before January 10, 2021, regarding artificial intelligence applications in diagnosing and predicting erectile dysfunction, were retrieved from five databases, including PubMed, EMBASE, the Cochrane Library, and two Chinese databases (WANFANG and CNKI). In addition, the reference lists of the included studies or relevant reviews were checked to avoid bias. Finally, 30 articles were reviewed to summarize the current status, merits, and limitations of applying artificial intelligence in diagnosing and predicting erectile dysfunction. The results showed that artificial intelligence contributed to developing novel diagnostic questionnaires, equipment, expert systems, classifiers by images and predictive models. However, most of the included studies were not subjected to external validations, resulting in doubt on the generalizability. In the future, more rigorously designed studies with high-quality datasets for erectile dysfunction are required.

Prediction and Diagnosis of Venous Thromboembolism Using Artificial Intelligence Approaches: A Systematic Review and Meta-Analysis

Venous thromboembolism (VTE) is a fatal disease and has become a burden on the global health system. Recent studies have suggested that artificial intelligence (AI) could be used to make a diagnosis and predict venous thrombosis more accurately. Thus, we performed a meta-analysis to better evaluate the performance of AI in the prediction and diagnosis of venous thrombosis. PubMed, Web of Science, and EMBASE were used to identify relevant studies. Of the 741 studies, 12 met the inclusion criteria and were included in the meta-analysis. Among them, 5 studies included a training set and test set, and 7 studies included only a training set. In the training set, the pooled sensitivity was 0.87 (95% CI 0.79-0.92), the pooled specificity was 0.95 (95% CI 0.89-0.97), and the area under the summary receiver operating characteristic (SROC) curve was 0.97 (95% CI 0.95-0.98). In the test set, the pooled sensitivity was 0.87 (95% CI 0.74-0.93), the pooled specificity was 0.96 (95% CI 0.79-0.99), and the area under the SROC curve was 0.98 (95% CI 0.97-0.99). The combined results remained significant in the subgroup analyzes, which included venous thrombosis type, AI type, model type (diagnosis/prediction), and whether the period was perioperative. In conclusion, AI may aid in the diagnosis and prediction of venous thrombosis, demonstrating high sensitivity, specificity and area under the SROC curve values. Thus, AI has important clinical value.

Natural language processing for the surveillance of postoperative venous thromboembolism

BACKGROUND

The objective of the study was to develop a portal natural language processing approach to aid in the identification of postoperative venous thromboembolism events from free-text clinical notes.

METHODS

We abstracted clinical notes from 25,494 operative events from 2 independent health care systems. A venous thromboembolism detected as part of the American College of Surgeons National Surgical Quality Improvement Program was used as the reference standard. A natural language processing engine, easy clinical information extractor-pulmonary embolism/deep vein thrombosis (EasyCIE-PEDVT), was trained to detect pulmonary embolism and deep vein thrombosis from clinical notes. International Classification of Diseases discharge diagnosis codes for venous thromboembolism were used as baseline comparators. The classification performance of EasyCIE-PEDVT was compared with International Classification of Diseases codes using sensitivity, specificity, area under the receiver operating characteristic curve, using an internal and external validation cohort.

RESULTS

To detect pulmonary embolism, EasyCIE-PEDVT had a sensitivity of 0.714 and 0.815 in internal and external validation, respectively. To detect deep vein thrombosis, EasyCIE-PEDVT had a sensitivity of 0.846 and 0.849 in internal and external validation, respectively. EasyCIE-PEDVT had significantly higher discrimination for deep vein thrombosis compared with International Classification of Diseases codes in internal validation (area under the receiver operating characteristic curve: 0.920 vs 0.761; P < .001) and external validation (area under the receiver operating characteristic curve: 0.921 vs 0.794; P < .001). There was no significant difference in the discrimination for pulmonary embolism between EasyCIE-PEDVT and International Classification of Diseases codes.

CONCLUSION

Accurate surveillance of postoperative venous thromboembolism may be achieved using natural language processing on clinical notes in 2 independent health care systems. These findings suggest natural language processing may augment manual chart abstraction for large registries such as National Surgical Quality Improvement Program.

Natural Language Processing in Surgery: A Systematic Review and Meta-analysis

OBJECTIVE

The aim of this study was to systematically assess the application and potential benefits of natural language processing (NLP) in surgical outcomes research.

SUMMARY BACKGROUND DATA

Widespread implementation of electronic health records (EHRs) has generated a massive patient data source. Traditional methods of data capture, such as billing codes and/or manual review of free-text narratives in EHRs, are highly labor-intensive, costly, subjective, and potentially prone to bias.

METHODS

A literature search of PubMed, MEDLINE, Web of Science, and Embase identified all articles published starting in 2000 that used NLP models to assess perioperative surgical outcomes. Evaluation metrics of NLP systems were assessed by means of pooled analysis and meta-analysis. Qualitative synthesis was carried out to assess the results and risk of bias on outcomes.

RESULTS

The present study included 29 articles, with over half (n = 15) published after 2018. The most common outcome identified using NLP was postoperative complications (n = 14). Compared to traditional non-NLP models, NLP models identified postoperative complications with higher sensitivity [0.92 (0.87-0.95) vs 0.58 (0.33-0.79), P < 0.001]. The specificities were comparable at 0.99 (0.96-1.00) and 0.98 (0.95-0.99), respectively. Using summary of likelihood ratio matrices, traditional non-NLP models have clinical utility for confirming documentation of outcomes/diagnoses, whereas NLP models may be reliably utilized for both confirming and ruling out documentation of outcomes/diagnoses.

CONCLUSIONS

NLP usage to extract a range of surgical outcomes, particularly postoperative complications, is accelerating across disciplines and areas of clinical outcomes research. NLP and traditional non-NLP approaches demonstrate similar performance measures, but NLP is superior in ruling out documentation of surgical outcomes.

Validation of a Semiautomated Natural Language Processing-Based Procedure for Meta-Analysis of Cancer Susceptibility Gene Penetrance

PURPOSE

Quantifying the risk of cancer associated with pathogenic mutations in germline cancer susceptibility genes-that is, penetrance-enables the personalization of preventive management strategies. Conducting a meta-analysis is the best way to obtain robust risk estimates. We have previously developed a natural language processing (NLP) -based abstract classifier which classifies abstracts as relevant to penetrance, prevalence of mutations, both, or neither. In this work, we evaluate the performance of this NLP-based procedure.

MATERIALS AND METHODS

We compared the semiautomated NLP-based procedure, which involves automated abstract classification and text mining, followed by human review of identified studies, with the traditional procedure that requires human review of all studies. Ten high-quality gene-cancer penetrance meta-analyses spanning 16 gene-cancer associations were used as the gold standard by which to evaluate the performance of our procedure. For each meta-analysis, we evaluated the number of abstracts that required human review (workload) and the ability to identify the studies that were included by the authors in their quantitative analysis (coverage).

RESULTS

Compared with the traditional procedure, the semiautomated NLP-based procedure led to a lower workload across all 10 meta-analyses, with an overall 84% reduction (2,774 abstracts v 16,941 abstracts) in the amount of human review required. Overall coverage was 93%-we are able to identify 132 of 142 studies-before reviewing references of identified studies. Reasons for the 10 missed studies included blank and poorly written abstracts. After reviewing references, nine of the previously missed studies were identified and coverage improved to 99% (141 of 142 studies).

CONCLUSION

We demonstrated that an NLP-based procedure can significantly reduce the review workload without compromising the ability to identify relevant studies. NLP algorithms have promising potential for reducing human efforts in the literature review process.