Assessment of Deep Natural Language Processing in Ascertaining Oncologic Outcomes From Radiology Reports

Identification of Child Survivors of Sex Trafficking From Electronic Health Records: An Artificial Intelligence Guided Approach

Survivors of child sex trafficking (SCST) experience high rates of adverse health outcomes. Amidst the duration of their victimization, survivors regularly seek healthcare yet fail to be identified. This study sought to utilize artificial intelligence (AI) to identify SCST and describe the elements of their healthcare presentation. An AI-supported keyword search was conducted to identify SCST within the electronic medical records (EMR) of ∼1.5 million patients at a large midwestern pediatric hospital. Descriptive analyses were used to evaluate associated diagnoses and clinical presentation. A sex trafficking-related keyword was identified in .18% of patient charts. Among this cohort, the most common associated diagnostic codes were for Confirmed Sexual/Physical Assault; Trauma and Stress-Related Disorders; Depressive Disorders; Anxiety Disorders; and Suicidal Ideation. Our findings are consistent with the myriad of known adverse physical and psychological outcomes among SCST and illuminate the future potential of AI technology to improve screening and research efforts surrounding all aspects of this vulnerable population.

Molecular Characteristics and Pretreatment Neutrophil-to-Lymphocyte Ratio as Predictors of Durable Clinical Benefit from Immune Checkpoint Inhibition in Non-Small Cell Lung Cancer

BACKGROUND

Prior research in non-small cell lung cancer (NSCLC) has shown that tumors with specific driver mutations may be less likely to respond to immune checkpoint inhibitors (ICI). In this analysis, we evaluated the characteristics of patients with durable clinical benefit (DCB) to ICI compared to those with no durable clinical benefit (NDB), with emphasis on the role of molecular alterations in EGFR, ALK, and ROS1 and pretreatment neutrophil-to-lymphocyte ratio (NLR).

METHODS

We retrospectively collected clinical characteristics and outcomes for patients who initiated ICI monotherapy for advanced NSCLC at Stanford University between April 2015 and May 2018. Patients were classified as having DCB if time on ICI therapy was greater than or equal to 180 days, or NDB if less than 180 days. Outcomes included best radiographic benefit while on ICI and survival from time of ICI initiation.

RESULTS

Of 123 patients treated with ICI for NSCLC, 28 patients had DCB (23%), while 95 had NDB (77%). Median overall survival from initiation of ICI in the 33 patients with molecular alterations in EGFR (n = 31), ALK, or ROS1 and NLR of 5.9 or higher was 2.0 months, compared to 8.1 months in patients with these genomic alterations and NLR less than 5.9. Median overall survival in patients without alterations in EGFR, ALK, or ROS1 and NLR of 5.9 or higher was 4.3 months, compared to 12.1 months in patients with NLR less than 5.9 (P = .023).

CONCLUSIONS

Elevation in pretreatment NLR was associated with significantly lower overall median survival from initiation of ICI, particularly when in combination with NSCLC with alterations in EGFR, ALK, or ROS1. This finding could influence clinical practice as NLR is readily available through routine blood testing.

Potential application of artificial intelligence in cancer therapy

PURPOSE OF REVIEW

This review underscores the critical role and challenges associated with the widespread adoption of artificial intelligence in cancer care to enhance disease management, streamline clinical processes, optimize data retrieval of health information, and generate and synthesize evidence.

RECENT FINDINGS

Advancements in artificial intelligence models and the development of digital biomarkers and diagnostics are applicable across the cancer continuum from early detection to survivorship care. Additionally, generative artificial intelligence has promised to streamline clinical documentation and patient communications, generate structured data for clinical trial matching, automate cancer registries, and facilitate advanced clinical decision support. Widespread adoption of artificial intelligence has been slow because of concerns about data diversity and data shift, model reliability and algorithm bias, legal oversight, and high information technology and infrastructure costs.

SUMMARY

Artificial intelligence models have significant potential to transform cancer care. Efforts are underway to deploy artificial intelligence models in the cancer practice, evaluate their clinical impact, and enhance their fairness and explainability. Standardized guidelines for the ethical integration of artificial intelligence models in cancer care pathways and clinical operations are needed. Clear governance and oversight will be necessary to gain trust in artificial intelligence-assisted cancer care by clinicians, scientists, and patients.

Use of Natural Language Processing to Extract and Classify Papillary Thyroid Cancer Features From Surgical Pathology Reports

BACKGROUND

We aim to use Natural Language Processing to automate the extraction and classification of thyroid cancer risk factors from pathology reports.

METHODS

We analyzed 1410 surgical pathology reports from adult papillary thyroid cancer patients from 2010 to 2019. Structured and nonstructured reports were used to create a consensus-based ground truth dictionary and categorized them into modified recurrence risk levels. Nonstructured reports were narrative, while structured reports followed standardized formats. We developed ThyroPath, a rule-based Natural Language Processing pipeline, to extract and classify thyroid cancer features into risk categories. Training involved 225 reports (150 structured, 75 unstructured), with testing on 170 reports (120 structured, 50 unstructured) for evaluation. The pipeline's performance was assessed using both strict and lenient criteria for accuracy, precision, recall, and F1-score; a metric that combines precision and recall evaluation.

RESULTS

In extraction tasks, ThyroPath achieved overall strict F-1 scores of 93% for structured reports and 90% for unstructured reports, covering 18 thyroid cancer pathology features. In classification tasks, ThyroPath-extracted information demonstrated an overall accuracy of 93% in categorizing reports based on their corresponding guideline-based risk of recurrence: 76.9% for high-risk, 86.8% for intermediate risk, and 100% for both low and very low-risk cases. However, ThyroPath achieved 100% accuracy across all risk categories with human extracted pathology information.

CONCLUSIONS

ThyroPath shows promise in automating the extraction and risk recurrence classification of thyroid pathology reports at large scale. It offers a solution to laborious manual reviews and advancing virtual registries. However, it requires further validation before implementation.

The Fine-Tuned Large Language Model for Extracting the Progressive Bone Metastasis from Unstructured Radiology Reports

Early detection of patients with impending bone metastasis is crucial for prognosis improvement. This study aimed to investigate the feasibility of a fine-tuned, locally run large language model (LLM) in extracting patients with bone metastasis in unstructured Japanese radiology report and to compare its performance with manual annotation. This retrospective study included patients with "metastasis" in radiological reports (April 2018-January 2019, August-May 2022, and April-December 2023 for training, validation, and test datasets of 9559, 1498, and 7399 patients, respectively). Radiologists reviewed the clinical indication and diagnosis sections of the radiological report (used as input data) and classified them into groups 0 (no bone metastasis), 1 (progressive bone metastasis), and 2 (stable or decreased bone metastasis). The data for group 0 was under-sampled in training and test datasets due to group imbalance. The best-performing model from the validation set was subsequently tested using the testing dataset. Two additional radiologists (readers 1 and 2) were involved in classifying radiological reports within the test dataset for testing purposes. The fine-tuned LLM, reader 1, and reader 2 demonstrated an accuracy of 0.979, 0.996, and 0.993, sensitivity for groups 0/1/2 of 0.988/0.947/0.943, 1.000/1.000/0.966, and 1.000/0.982/0.954, and time required for classification (s) of 105, 2312, and 3094 in under-sampled test dataset (n = 711), respectively. Fine-tuned LLM extracted patients with bone metastasis, demonstrating satisfactory performance that was comparable to or slightly lower than manual annotation by radiologists in a noticeably shorter time.

Knowledge domain and emerging trends in the rupture risk of intracranial aneurysms research from 2004 to 2023

BACKGROUND

Intracranial aneurysms (IAs) pose significant health risks, attributable to their potential for sudden rupture, which can result in severe outcomes such as stroke and death. Despite extensive research, the variability of aneurysm behavior, with some remaining stable for years while others rupture unexpectedly, remains poorly understood.

AIM

To employ bibliometric analysis to map the research landscape concerning risk factors associated with IAs rupture.

METHODS

A systematic literature review of publications from 2004 to 2023 was conducted, analyzing 3804 documents from the Web of Science Core Collection database, with a focus on full-text articles and reviews in English. The analysis encompassed citation and co-citation networks, keyword bursts, and temporal trends to delineate the evolution of research themes and collaboration patterns. Advanced software tools, CiteSpace and VOSviewer, were utilized for comprehensive data visualization and trend analysis.

RESULTS

Analysis uncovered a total of 3804 publications on IA rupture risk factors between 2006 and 2023. Research interest surged after 2013, peaking in 2023. The United States led with 28.97% of publications, garnering 37706 citations. Notable United States-China collaborations were observed. Capital Medical University produced 184 publications, while Utrecht University boasted a citation average of 69.62 per publication. "World Neurosurgery" published the most papers, contrasting with "Stroke", the most cited journal. The PHASES score from "Lancet Neurology" emerged as a vital rupture risk prediction tool. Early research favored endovascular therapy, transitioning to magnetic resonance imaging and flow diverters. "Subarachnoid hemorrhage" stood out as a recurrent keyword.

CONCLUSION

This study assesses global IA research trends and highlights crucial gaps, guiding future investigations to improve preventive and therapeutic approaches.

Automated classification of brain MRI reports using fine-tuned large language models

PURPOSE

This study aimed to investigate the efficacy of fine-tuned large language models (LLM) in classifying brain MRI reports into pretreatment, posttreatment, and nontumor cases.

METHODS

This retrospective study included 759, 284, and 164 brain MRI reports for training, validation, and test dataset. Radiologists stratified the reports into three groups: nontumor (group 1), posttreatment tumor (group 2), and pretreatment tumor (group 3) cases. A pretrained Bidirectional Encoder Representations from Transformers Japanese model was fine-tuned using the training dataset and evaluated on the validation dataset. The model which demonstrated the highest accuracy on the validation dataset was selected as the final model. Two additional radiologists were involved in classifying reports in the test datasets for the three groups. The model's performance on test dataset was compared to that of two radiologists.

RESULTS

The fine-tuned LLM attained an overall accuracy of 0.970 (95% CI: 0.930-0.990). The model's sensitivity for group 1/2/3 was 1.000/0.864/0.978. The model's specificity for group1/2/3 was 0.991/0.993/0.958. No statistically significant differences were found in terms of accuracy, sensitivity, and specificity between the LLM and human readers (p ≥ 0.371). The LLM completed the classification task approximately 20-26-fold faster than the radiologists. The area under the receiver operating characteristic curve for discriminating groups 2 and 3 from group 1 was 0.994 (95% CI: 0.982-1.000) and for discriminating group 3 from groups 1 and 2 was 0.992 (95% CI: 0.982-1.000).

CONCLUSION

Fine-tuned LLM demonstrated a comparable performance with radiologists in classifying brain MRI reports, while requiring substantially less time.

Extraction of Unstructured Electronic Health Records to Evaluate Glioblastoma Treatment Patterns

PURPOSE

Data on lines of therapy (LOTs) for cancer treatment are important for clinical oncology research, but LOTs are not explicitly recorded in electronic health records (EHRs). We present an efficient approach for clinical data abstraction and a flexible algorithm to derive LOTs from EHR-based medication data on patients with glioblastoma multiforme (GBM).

METHODS

Nonclinicians were trained to abstract the diagnosis of GBM from EHRs, and their accuracy was compared with abstraction performed by clinicians. The resulting data were used to build a cohort of patients with confirmed GBM diagnosis. An algorithm was developed to derive LOTs using structured medication data, accounting for the addition and discontinuation of therapies and drug class. Descriptive statistics were calculated and time-to-next-treatment (TTNT) analysis was performed using the Kaplan-Meier method.

RESULTS

Treating clinicians as the gold standard, nonclinicians abstracted GBM diagnosis with a sensitivity of 0.98, specificity 1.00, positive predictive value 1.00, and negative predictive value 0.90, suggesting that nonclinician abstraction of GBM diagnosis was comparable with clinician abstraction. Of 693 patients with a confirmed diagnosis of GBM, 246 patients contained structured information about the types of medications received. Of them, 165 (67.1%) received a first-line therapy (1L) of temozolomide, and the median TTNT from the start of 1L was 179 days.

CONCLUSION

We described a workflow for extracting diagnosis of GBM and LOT from EHR data that combines nonclinician abstraction with algorithmic processing, demonstrating comparable accuracy with clinician abstraction and highlighting the potential for scalable and efficient EHR-based oncology research.

Natural language processing augments comorbidity documentation in neurosurgical inpatient admissions

OBJECTIVE

To establish whether or not a natural language processing technique could identify two common inpatient neurosurgical comorbidities using only text reports of inpatient head imaging.

MATERIALS AND METHODS

A training and testing dataset of reports of 979 CT or MRI scans of the brain for patients admitted to the neurosurgery service of a single hospital in June 2021 or to the Emergency Department between July 1-8, 2021, was identified. A variety of machine learning and deep learning algorithms utilizing natural language processing were trained on the training set (84% of the total cohort) and tested on the remaining images. A subset comparison cohort (n = 76) was then assessed to compare output of the best algorithm against real-life inpatient documentation.

RESULTS

For "brain compression", a random forest classifier outperformed other candidate algorithms with an accuracy of 0.81 and area under the curve of 0.90 in the testing dataset. For "brain edema", a random forest classifier again outperformed other candidate algorithms with an accuracy of 0.92 and AUC of 0.94 in the testing dataset. In the provider comparison dataset, for "brain compression," the random forest algorithm demonstrated better accuracy (0.76 vs 0.70) and sensitivity (0.73 vs 0.43) than provider documentation. For "brain edema," the algorithm again demonstrated better accuracy (0.92 vs 0.84) and AUC (0.45 vs 0.09) than provider documentation.

DISCUSSION

A natural language processing-based machine learning algorithm can reliably and reproducibly identify selected common neurosurgical comorbidities from radiology reports.

CONCLUSION

This result may justify the use of machine learning-based decision support to augment provider documentation.

Use of Natural Language Processing to Infer Sites of Metastatic Disease From Radiology Reports at Scale

PURPOSE

To evaluate natural language processing (NLP) methods to infer metastatic sites from radiology reports.

METHODS

A set of 4,522 computed tomography (CT) reports of 550 patients with 14 types of cancer was used to fine-tune four clinical large language models (LLMs) for multilabel classification of metastatic sites. We also developed an NLP information extraction (IE) system (on the basis of named entity recognition, assertion status detection, and relation extraction) for comparison. Model performances were measured by F1 scores on test and three external validation sets. The best model was used to facilitate analysis of metastatic frequencies in a cohort study of 6,555 patients with 53,838 CT reports.

RESULTS

The RadBERT, BioBERT, GatorTron-base, and GatorTron-medium LLMs achieved F1 scores of 0.84, 0.87, 0.89, and 0.91, respectively, on the test set. The IE system performed best, achieving an F1 score of 0.93. F1 scores of the IE system by individual cancer type ranged from 0.89 to 0.96. The IE system attained F1 scores of 0.89, 0.83, and 0.81, respectively, on external validation sets including additional cancer types, positron emission tomography-CT ,and magnetic resonance imaging scans, respectively. In our cohort study, we found that for colorectal cancer, liver-only metastases were higher in de novo stage IV versus recurrent patients (29.7% v 12.2%; P < .001). Conversely, lung-only metastases were more frequent in recurrent versus de novo stage IV patients (17.2% v 7.3%; P < .001).

CONCLUSION

We developed an IE system that accurately infers metastatic sites in multiple primary cancers from radiology reports. It has explainable methods and performs better than some clinical LLMs. The inferred metastatic phenotypes could enhance cancer research databases and clinical trial matching, and identify potential patients for oligometastatic interventions.

Extraction and Imputation of Eastern Cooperative Oncology Group Performance Status From Unstructured Oncology Notes Using Language Models

PURPOSE

Eastern Cooperative Oncology Group (ECOG) performance status (PS) is a key clinical variable for cancer treatment and research, but it is usually only recorded in unstructured form in the electronic health record. We investigated whether natural language processing (NLP) models can impute ECOG PS using unstructured note text.

MATERIALS AND METHODS

Medical oncology notes were identified from all patients with cancer at our center from 1997 to 2023 and divided at the patient level into training (approximately 80%), tuning/validation (approximately 10%), and test (approximately 10%) sets. Regular expressions were used to extract explicitly documented PS. Extracted PS labels were used to train NLP models to impute ECOG PS (0-1 v 2-4) from the remainder of the notes (with regular expression-extracted PS documentation removed). We assessed associations between imputed PS and overall survival (OS).

RESULTS

ECOG PS was extracted using regular expressions from 495,862 notes, corresponding to 79,698 patients. A Transformer-based Longformer model imputed PS with high discrimination (test set area under the receiver operating characteristic curve 0.95, area under the precision-recall curve 0.73). Imputed poor PS was associated with worse OS, including among notes with no explicit documentation of PS detected (OS hazard ratio, 11.9; 95% CI, 11.1 to 12.8).

CONCLUSION

NLP models can be used to impute performance status from unstructured oncologist notes at scale. This may aid the annotation of oncology data sets for clinical outcomes research and cancer care delivery.

Development and Operationalization of an Automated Workflow for Correlation of Knee MRI and Arthroscopy Findings

OBJECTIVE

In this study, we sought to establish and evaluate an automated workflow to prospectively capture and correlate knee MRI findings with surgical findings in a large medical center.

METHODS

This retrospective analysis included data from patients who had undergone knee MRI followed by arthroscopic knee surgery within 6 months during a 2-year period (2019-2020). Discrete data were automatically extracted from a structured knee MRI report template implementing pick lists. Operative findings were recorded discretely by surgeons using a custom-built web-based telephone application. MRI findings were classified as true-positive, true-negative, false-positive, or false-negative for medial meniscus (MM), lateral meniscus (LM), and anterior cruciate ligament (ACL) tears, with arthroscopy used as the reference standard. An automated dashboard displaying up-to-date concordance and individual and group accuracy was enabled for each radiologist. Manual correlation between MRI and operative reports was performed on a random sample of 10% of cases for comparison with automatically derived values.

RESULTS

Data from 3,187 patients (1,669 male; mean age, 47 years) were analyzed. Automatic correlation was available for 60% of cases, with an overall MRI diagnostic accuracy of 93% (MM, 92%; LM, 89%; ACL, 98%). In cases reviewed manually, the number of cases that could be correlated with surgery was higher (84%). Concordance between automated and manual review was 99% when both were available (MM, 98%; LM, 100%; ACL, 99%).

CONCLUSION

This automated system was able to accurately and continuously assess correlation between imaging and operative findings for a large number of MRI examinations.

Biomarker Inference and the Timing of Next-Generation Sequencing in a Multi-Institutional, Cross-Cancer Clinicogenomic Data Set

PURPOSE

Observational clinicogenomic data sets, consisting of tumor next-generation sequencing (NGS) data linked to clinical records, are commonly used for cancer research. However, in real-world practice, oncologists frequently request NGS in search of treatment options for progressive cancer. The extent and impact of this dynamic on analysis of clinicogenomic research data are not well understood.

METHODS

We analyzed clinicogenomic data for patients with non-small cell lung, colorectal, breast, prostate, pancreatic, or urothelial cancers in the American Association for Cancer Research Biopharmaceutical Consortium cohort. Associations between baseline and time-varying clinical characteristics and time from diagnosis to NGS were measured. To explore the impact of informative cohort entry on biomarker inference, statistical interactions between selected biomarkers and time to NGS with respect to overall survival were calculated.

RESULTS

Among 7,182 patients, time from diagnosis to NGS varied significantly by clinical factors, including cancer type, calendar year of sequencing, institution, and age and stage at diagnosis. NGS rates also varied significantly by dynamic clinical status variables; in an adjusted model, compared with patients with stable disease at any given time after diagnosis, patients with progressive disease by imaging or oncologist assessment had higher NGS rates (hazard ratio for NGS, 1.61 [95% CI, 1.45 to 1.78] and 2.32 [95% CI, 2.01 to 2.67], respectively). Statistical interactions between selected biomarkers and time to NGS with respect to survival, potentially indicating biased biomarker inference results, were explored.

CONCLUSION

To evaluate the appropriateness of a data set for a particular research question, it is crucial to measure associations between dynamic cancer status and the timing of NGS, as well as to evaluate interactions involving biomarkers of interest and NGS timing with respect to survival outcomes.

Worldwide productivity and research trend of publications concerning intestinal polyps: A bibliometric study

There is a significant relationship between intestinal polyps and colorectal cancer, and in recent years, research on intestinal polyps has been rapidly developing around the world. However, there is still a lack of adequate quantification and analysis of publications in this field. The aim of this study was to perform a comprehensive bibliometric analysis of publications related to intestinal polyps over the past 20 years. To enhance the understanding of current research hotspots and potential trends, and to point out the direction of future research. Publications related to intestinal polyps were retrieved from the Science Citation Index Expanded in Web of Science Core Collection. the Bibliometric online analysis platform (https://bibliometric.com/app), the Bibliometrix Package, and the CiteSpace are used for bibliometric analysis and visualization, including the overall range of annual output and annual citations, country-region analysis, author and institution analysis, core journal analysis, reference and keyword analysis. Prior to 2017, the amount of research on intestinal polyps was slow to grow, but it picked up speed after that year. In 1019 journals, 4280 papers on intestinal polyps were published in English. The journal with the highest productivity was Gastrointestinal Endoscopy (189, 4.42%). United States (1124, 26.26%), which is also the hub of collaboration in this subject, was the most productive nation. Mayo Clinic (n = 70, 1.64%) is the most productive institution. Intestinal microbiota, endoscopic mucosal resection, gut microbiota, deep learning, tea polyphenol, insulin resistance and artificial intelligence were current hot subjects in the field. Studies of intestinal polyps increased significantly after 2017. The United States contributed the largest number of publications. Countries and institutions were actively cooperating with one another. artificial intelligence is currently an emerging topic.

Oncogenic Drivers and Therapeutic Vulnerabilities in KRAS Wild-Type Pancreatic Cancer

PURPOSE

Approximately 8% to 10% of pancreatic ductal adenocarcinomas (PDAC) do not harbor mutations in KRAS. Understanding the unique molecular and clinical features of this subset of pancreatic cancer is important to guide patient stratification for clinical trials of molecularly targeted agents.

EXPERIMENTAL DESIGN

We analyzed a single-institution cohort of 795 exocrine pancreatic cancer cases (including 785 PDAC cases) with a targeted multigene sequencing panel and identified 73 patients (9.2%) with KRAS wild-type (WT) pancreatic cancer.

RESULTS

Overall, 43.8% (32/73) of KRAS WT cases had evidence of an alternative driver of the MAPK pathway, including BRAF mutations and in-frame deletions and receptor tyrosine kinase fusions. Conversely, 56.2% of cases did not harbor a clear MAPK driver alteration, but 29.3% of these MAPK-negative KRAS WT cases (12/41) demonstrated activating alterations in other oncogenic drivers, such as GNAS, MYC, PIK3CA, and CTNNB1. We demonstrate potent efficacy of pan-RAF and MEK inhibition in patient-derived organoid models carrying BRAF in-frame deletions. Moreover, we demonstrate durable clinical benefit of targeted therapy in a patient harboring a KRAS WT tumor with a ROS1 fusion. Clinically, patients with KRAS WT tumors were significantly younger in age of onset (median age: 62.6 vs. 65.7 years; P = 0.037). SMAD4 mutations were associated with a particularly poor prognosis in KRAS WT cases.

CONCLUSIONS

This study defines the genomic underpinnings of KRAS WT pancreatic cancer and highlights potential therapeutic avenues for future investigation in molecularly directed clinical trials. See related commentary by Kato et al., p. 4527.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.

Overview of approaches to estimate real-world disease progression in lung cancer

BACKGROUND

Randomized clinical trials of novel treatments for solid tumors normally measure disease progression using the Response Evaluation Criteria in Solid Tumors. However, novel, scalable approaches to estimate disease progression using real-world data are needed to advance cancer outcomes research. The purpose of this narrative review is to summarize examples from the existing literature on approaches to estimate real-world disease progression and their relative strengths and limitations, using lung cancer as a case study.

METHODS

A narrative literature review was conducted in PubMed to identify articles that used approaches to estimate real-world disease progression in lung cancer patients. Data abstracted included data source, approach used to estimate real-world progression, and comparison to a selected gold standard (if applicable).

RESULTS

A total of 40 articles were identified from 2008 to 2022. Five approaches to estimate real-world disease progression were identified including manual abstraction of medical records, natural language processing of clinical notes and/or radiology reports, treatment-based algorithms, changes in tumor volume, and delta radiomics-based approaches. The accuracy of these progression approaches were assessed using different methods, including correlations between real-world endpoints and overall survival for manual abstraction (Spearman rank ρ = 0.61-0.84) and area under the curve for natural language processing approaches (area under the curve = 0.86-0.96).

CONCLUSIONS

Real-world disease progression has been measured in several observational studies of lung cancer. However, comparing the accuracy of methods across studies is challenging, in part, because of the lack of a gold standard and the different methods used to evaluate accuracy. Concerted efforts are needed to define a gold standard and quality metrics for real-world data.

The diagnosis of tuberculous meningitis: advancements in new technologies and machine learning algorithms

Tuberculous meningitis (TBM) poses a diagnostic challenge, particularly impacting vulnerable populations such as infants and those with untreated HIV. Given the diagnostic intricacies of TBM, there's a pressing need for rapid and reliable diagnostic tools. This review scrutinizes the efficacy of up-and-coming technologies like machine learning in transforming TBM diagnostics and management. Advanced diagnostic technologies like targeted gene sequencing, real-time polymerase chain reaction (RT-PCR), miRNA assays, and metagenomic next-generation sequencing (mNGS) offer promising avenues for early TBM detection. The capabilities of these technologies are further augmented when paired with mass spectrometry, metabolomics, and proteomics, enriching the pool of disease-specific biomarkers. Machine learning algorithms, adept at sifting through voluminous datasets like medical imaging, genomic profiles, and patient histories, are increasingly revealing nuanced disease pathways, thereby elevating diagnostic accuracy and guiding treatment strategies. While these burgeoning technologies offer hope for more precise TBM diagnosis, hurdles remain in terms of their clinical implementation. Future endeavors should zero in on the validation of these tools through prospective studies, critically evaluating their limitations, and outlining protocols for seamless incorporation into established healthcare frameworks. Through this review, we aim to present an exhaustive snapshot of emerging diagnostic modalities in TBM, the current standing of machine learning in meningitis diagnostics, and the challenges and future prospects of converging these domains.

Applying Natural Language Processing to Single-Report Prediction of Metastatic Disease Response Using the OR-RADS Lexicon

Generating Real World Evidence (RWE) on disease responses from radiological reports is important for understanding cancer treatment effectiveness and developing personalized treatment. A lack of standardization in reporting among radiologists impacts the feasibility of large-scale interpretation of disease response. This study examines the utility of applying natural language processing (NLP) to the large-scale interpretation of disease responses using a standardized oncologic response lexicon (OR-RADS) to facilitate RWE collection. Radiologists annotated 3503 retrospectively collected clinical impressions from radiological reports across several cancer types with one of seven OR-RADS categories. A Bidirectional Encoder Representations from Transformers (BERT) model was trained on this dataset with an 80-20% train/test split to perform multiclass and single-class classification tasks using the OR-RADS. Radiologists also performed the classification to compare human and model performance. The model achieved accuracies from 95 to 99% across all classification tasks, performing better in single-class tasks compared to the multiclass task and producing minimal misclassifications, which pertained mostly to overpredicting the equivocal and mixed OR-RADS labels. Human accuracy ranged from 74 to 93% across all classification tasks, performing better on single-class tasks. This study demonstrates the feasibility of the BERT NLP model in predicting disease response in cancer patients, exceeding human performance, and encourages the use of the standardized OR-RADS lexicon to improve large-scale prediction accuracy.

Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting

OBJECTIVE

To assess large language models on their ability to accurately infer cancer disease response from free-text radiology reports.

MATERIALS AND METHODS

We assembled 10 602 computed tomography reports from cancer patients seen at a single institution. All reports were classified into: no evidence of disease, partial response, stable disease, or progressive disease. We applied transformer models, a bidirectional long short-term memory model, a convolutional neural network model, and conventional machine learning methods to this task. Data augmentation using sentence permutation with consistency loss as well as prompt-based fine-tuning were used on the best-performing models. Models were validated on a hold-out test set and an external validation set based on Response Evaluation Criteria in Solid Tumors (RECIST) classifications.

RESULTS

The best-performing model was the GatorTron transformer which achieved an accuracy of 0.8916 on the test set and 0.8919 on the RECIST validation set. Data augmentation further improved the accuracy to 0.8976. Prompt-based fine-tuning did not further improve accuracy but was able to reduce the number of training reports to 500 while still achieving good performance.

DISCUSSION

These models could be used by researchers to derive progression-free survival in large datasets. It may also serve as a decision support tool by providing clinicians an automated second opinion of disease response.

CONCLUSIONS

Large clinical language models demonstrate potential to infer cancer disease response from radiology reports at scale. Data augmentation techniques are useful to further improve performance. Prompt-based fine-tuning can significantly reduce the size of the training dataset.

Empirical evaluation of language modeling to ascertain cancer outcomes from clinical text reports

BACKGROUND

Longitudinal data on key cancer outcomes for clinical research, such as response to treatment and disease progression, are not captured in standard cancer registry reporting. Manual extraction of such outcomes from unstructured electronic health records is a slow, resource-intensive process. Natural language processing (NLP) methods can accelerate outcome annotation, but they require substantial labeled data. Transfer learning based on language modeling, particularly using the Transformer architecture, has achieved improvements in NLP performance. However, there has been no systematic evaluation of NLP model training strategies on the extraction of cancer outcomes from unstructured text.

RESULTS

We evaluated the performance of nine NLP models at the two tasks of identifying cancer response and cancer progression within imaging reports at a single academic center among patients with non-small cell lung cancer. We trained the classification models under different conditions, including training sample size, classification architecture, and language model pre-training. The training involved a labeled dataset of 14,218 imaging reports for 1112 patients with lung cancer. A subset of models was based on a pre-trained language model, DFCI-ImagingBERT, created by further pre-training a BERT-based model using an unlabeled dataset of 662,579 reports from 27,483 patients with cancer from our center. A classifier based on our DFCI-ImagingBERT, trained on more than 200 patients, achieved the best results in most experiments; however, these results were marginally better than simpler "bag of words" or convolutional neural network models.

CONCLUSION

When developing AI models to extract outcomes from imaging reports for clinical cancer research, if computational resources are plentiful but labeled training data are limited, large language models can be used for zero- or few-shot learning to achieve reasonable performance. When computational resources are more limited but labeled training data are readily available, even simple machine learning architectures can achieve good performance for such tasks.

Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?

OBJECTIVES

Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.

METHODS

We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.

RESULTS

In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.

CONCLUSIONS

With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better.

ADVANCES IN KNOWLEDGE

Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

Cancer Informatics for Cancer Centers: Sharing Ideas on How to Build an Artificial Intelligence-Ready Informatics Ecosystem for Radiation Oncology

In August 2022, the Cancer Informatics for Cancer Centers brought together cancer informatics leaders for its biannual symposium, Precision Medicine Applications in Radiation Oncology, co-chaired by Quynh-Thu Le, MD (Stanford University), and Walter J. Curran, MD (GenesisCare). Over the course of 3 days, presenters discussed a range of topics relevant to radiation oncology and the cancer informatics community more broadly, including biomarker development, decision support algorithms, novel imaging tools, theranostics, and artificial intelligence (AI) for the radiotherapy workflow. Since the symposium, there has been an impressive shift in the promise and potential for integration of AI in clinical care, accelerated in large part by major advances in generative AI. AI is now poised more than ever to revolutionize cancer care. Radiation oncology is a field that uses and generates a large amount of digital data and is therefore likely to be one of the first fields to be transformed by AI. As experts in the collection, management, and analysis of these data, the informatics community will take a leading role in ensuring that radiation oncology is prepared to take full advantage of these technological advances. In this report, we provide highlights from the symposium, which took place in Santa Barbara, California, from August 29 to 31, 2022. We discuss lessons learned from the symposium for data acquisition, management, representation, and sharing, and put these themes into context to prepare radiation oncology for the successful and safe integration of AI and informatics technologies.

Artificial Intelligence to Improve Patient Understanding of Radiology Reports

Diagnostic imaging reports are generally written with a target audience of other providers. As a result, the reports are written with medical jargon and technical detail to ensure accurate communication. With implementation of the 21st Century Cures Act, patients have greater and quicker access to their imaging reports, but these reports are still written above the comprehension level of the average patient. Consequently, many patients have requested reports to be conveyed in language accessible to them. Numerous studies have shown that improving patient understanding of their condition results in better outcomes, so driving comprehension of imaging reports is essential. Summary statements, second reports, and the inclusion of the radiologist's phone number have been proposed, but these solutions have implications for radiologist workflow. Artificial intelligence (AI) has the potential to simplify imaging reports without significant disruptions. Many AI technologies have been applied to radiology reports in the past for various clinical and research purposes, but patient focused solutions have largely been ignored. New natural language processing technologies and large language models (LLMs) have the potential to improve patient understanding of their imaging reports. However, LLMs are a nascent technology and significant research is required before LLM-driven report simplification is used in patient care.

Deep learning-based detection of patients with bone metastasis from Japanese radiology reports

PURPOSE

Deep learning (DL) is a state-of-the-art technique for developing artificial intelligence in various domains and it improves the performance of natural language processing (NLP). Therefore, we aimed to develop a DL-based NLP model that classifies the status of bone metastasis (BM) in radiology reports to detect patients with BM.

MATERIALS AND METHODS

The DL-based NLP model was developed by training long short-term memory using 1,749 free-text radiology reports written in Japanese. We adopted five-fold cross-validation and used 200 reports for testing the five models. The accuracy, sensitivity, specificity, precision, and area under the receiver operating characteristics curve (AUROC) were used for the model evaluation.

RESULTS

The developed model demonstrated classification performance with mean ± standard deviation of 0.912 ± 0.012, 0.924 ± 0.029, 0.901 ± 0.014, 0.898 ± 0.012, and 0.968 ± 0.004 for accuracy, sensitivity, specificity, precision, and AUROC, respectively.

CONCLUSION

The proposed DL-based NLP model may help in the early and efficient detection of patients with BM.

Clinical and Genomic Features of Response and Toxicity to Sotorasib in a Real-World Cohort of Patients With Advanced KRAS G12C-Mutant Non-Small Cell Lung Cancer

PURPOSE

With the recent approval of the KRAS G12C inhibitor sotorasib for patients with advanced KRAS G12C-mutant non-small cell lung cancer (NSCLC), there is a new need to identify factors associated with activity and toxicity among patients treated in routine practice.

MATERIALS AND METHODS

We conducted a multicenter retrospective study of patients treated with sotorasib outside of clinical trials to identify factors associated with real-world progression free survival (rwPFS), overall survival (OS), and toxicity.

RESULTS

Among 105 patients with advanced KRAS G12C-mutant NSCLC treated with sotorasib, treatment led to a 5.3-month median rwPFS, 12.6-month median OS, and 28% real-world response rate. KEAP1 comutations were associated with shorter rwPFS and OS (rwPFS hazard ratio [HR], 3.19; P = .004; OS HR, 4.10; P = .003); no significant differences in rwPFS or OS were observed across TP53 (rwPFS HR, 1.10; P = .731; OS HR, 1.19; P = .631) or STK11 (rwPFS HR, 1.66; P = .098; OS HR, 1.73; P = .168) comutation status. Notably, almost all patients who developed grade 3 or higher treatment-related adverse events (G3+ TRAEs) had previously been treated with anti-PD-(L)1 therapy. Among these patients, anti-PD-(L)1 therapy exposure within 12 weeks of sotorasib was strongly associated with G3+ TRAEs (P < .001) and TRAE-related sotorasib discontinuation (P = .014). Twenty-eight percent of patients with recent anti-PD-(L)1 therapy exposure experienced G3+ TRAEs, most commonly hepatotoxicity.

CONCLUSION

Among patients treated with sotorasib in routine practice, KEAP1 comutations were associated with resistance and recent anti-PD-(L)1 therapy exposure was associated with toxicity. These observations may help guide use of sotorasib in the clinic and may help inform the next generation of KRAS G12C-targeted clinical trials.

Leveraging electronic health records to streamline the conduct of cardiovascular clinical trials

Conventional randomized controlled trials (RCTs) can be expensive, time intensive, and complex to conduct. Trial recruitment, participation, and data collection can burden participants and research personnel. In the past two decades, there have been rapid technological advances and an exponential growth in digitized healthcare data. Embedding RCTs, including cardiovascular outcome trials, into electronic health record systems or registries may streamline screening, consent, randomization, follow-up visits, and outcome adjudication. Moreover, wearable sensors (i.e. health and fitness trackers) provide an opportunity to collect data on cardiovascular health and risk factors in unprecedented detail and scale, while growing internet connectivity supports the collection of patient-reported outcomes. There is a pressing need to develop robust mechanisms that facilitate data capture from diverse databases and guidance to standardize data definitions. Importantly, the data collection infrastructure should be reusable to support multiple cardiovascular RCTs over time. Systems, processes, and policies will need to have sufficient flexibility to allow interoperability between different sources of data acquisition. Clinical research guidelines, ethics oversight, and regulatory requirements also need to evolve. This review highlights recent progress towards the use of routinely generated data to conduct RCTs and discusses potential solutions for ongoing barriers. There is a particular focus on methods to utilize routinely generated data for trials while complying with regional data protection laws. The discussion is supported with examples of cardiovascular outcome trials that have successfully leveraged the electronic health record, web-enabled devices or administrative databases to conduct randomized trials.

Toward structuring real-world data: Deep learning for extracting oncology information from clinical text with patient-level supervision

Most detailed patient information in real-world data (RWD) is only consistently available in free-text clinical documents. Manual curation is expensive and time consuming. Developing natural language processing (NLP) methods for structuring RWD is thus essential for scaling real-world evidence generation. We propose leveraging patient-level supervision from medical registries, which are often readily available and capture key patient information, for general RWD applications. We conduct an extensive study on 135,107 patients from the cancer registry of a large integrated delivery network (IDN) comprising healthcare systems in five western US states. Our deep-learning methods attain test area under the receiver operating characteristic curve (AUROC) values of 94%-99% for key tumor attributes and comparable performance on held-out data from separate health systems and states. Ablation results demonstrate the superiority of these advanced deep-learning methods. Error analysis shows that our NLP system sometimes even corrects errors in registrar labels.

Developing Artificial Intelligence Models for Extracting Oncologic Outcomes from Japanese Electronic Health Records

INTRODUCTION

A framework that extracts oncological outcomes from large-scale databases using artificial intelligence (AI) is not well established. Thus, we aimed to develop AI models to extract outcomes in patients with lung cancer using unstructured text data from electronic health records of multiple hospitals.

METHODS

We constructed AI models (Bidirectional Encoder Representations from Transformers [BERT], Naïve Bayes, and Longformer) for tumor evaluation using the University of Miyazaki Hospital (UMH) database. This data included both structured and unstructured data from progress notes, radiology reports, and discharge summaries. The BERT model was applied to the Life Data Initiative (LDI) data set of six hospitals. Study outcomes included the performance of AI models and time to progression of disease (TTP) for each line of treatment based on the treatment response extracted by AI models.

RESULTS

For the UMH data set, the BERT model exhibited higher precision accuracy compared to the Naïve Bayes or the Longformer models, respectively (precision [0.42 vs. 0.47 or 0.22], recall [0.63 vs. 0.46 or 0.33] and F1 scores [0.50 vs. 0.46 or 0.27]). When this BERT model was applied to LDI data, prediction accuracy remained quite similar. The Kaplan-Meier plots of TTP (months) showed similar trends for the first (median 14.9 [95% confidence interval 11.5, 21.1] and 16.8 [12.6, 21.8]), the second (7.8 [6.7, 10.7] and 7.8 [6.7, 10.7]), and the later lines of treatment for the predicted data by the BERT model and the manually curated data.

CONCLUSION

We developed AI models to extract treatment responses in patients with lung cancer using a large EHR database; however, the model requires further improvement.

Real-world treatment response in Japanese patients with cancer using unstructured data from electronic health records

Purpose

We generated methods for evaluating clinical outcomes including treatment response in oncology using the unstructured data from electronic health records (EHR) in Japanese language.

Methods

This retrospective analysis used medical record database and administrative data of University of Miyazaki Hospital in Japan of patients with lung/breast cancer. Treatment response (objective response [OR], stable disease [SD] or progressive disease [PD]) was adjudicated by two evaluators using clinicians’ progress notes, radiology reports and pathological reports of 15 patients with lung cancer (training data set). For assessing key terms to describe treatment response, natural language processing (NLP) rules were created from the texts identified by the evaluators and broken down by morphological analysis. The NLP rules were applied for assessing data of other 70 lung cancer and 30 breast cancer patients, who were not adjudicated, to examine if any difference in using key terms exist between these patients.

Results

A total of 2,039 records in progress notes, 131 in radiology reports and 60 in pathological reports of 15 patients, were adjudicated. Progress notes were the most common primary source data for treatment assessment (60.7%), wherein, the most common key terms with high sensitivity and specificity to describe OR were “reduction/shrink”, for SD were “(no) remarkable change/(no) aggravation)” and for PD were “(limited) effect” and “enlargement/grow”. These key terms were also found in other larger cohorts of 70 patients with lung cancer and 30 patients with breast cancer.

Conclusion

This study demonstrated that assessing response to anticancer therapy using Japanese EHRs is feasible by interpreting progress notes, radiology reports and Japanese key terms using NLP.

IKAR: An Interdisciplinary Knowledge-Based Automatic Retrieval Method from Chinese Electronic Medical Record

To date, information retrieval methods in the medical field have mainly focused on English medical reports, but little work has studied Chinese electronic medical reports, especially in the field of obstetrics and gynecology. In this paper, a dataset of 180,000 complete Chinese ultrasound reports in obstetrics and gynecology was established and made publicly available. Based on the ultrasound reports in the dataset, a new information retrieval method (IKAR) is proposed to extract key information from the ultrasound reports and automatically generate the corresponding ultrasound diagnostic results. The model can both extract what is already in the report and analyze what is not in the report by inference. After applying the IKAR method to the dataset, it is proved that the method could achieve 89.38% accuracy, 91.09% recall, and 90.23% F-score. Moreover, the method achieves an F-score of over 90% on 50% of the 10 components of the report. This study provides a quality dataset for the field of electronic medical records and offers a reference for information retrieval methods in the field of obstetrics and gynecology or in other fields.

Prediction of mortality risk and duration of hospitalization of COVID-19 patients with chronic comorbidities based on machine learning algorithms

Background

The severity of coronavirus (COVID-19) in patients with chronic comorbidities is much higher than in other patients, which can lead to their death. Machine learning (ML) algorithms as a potential solution for rapid and early clinical evaluation of the severity of the disease can help in allocating and prioritizing resources to reduce mortality.

Objective

The objective of this study was to predict the mortality risk and length of stay (LoS) of patients with COVID-19 and history of chronic comorbidities using ML algorithms.

Methods

This retrospective study was conducted by reviewing the medical records of COVID-19 patients with a history of chronic comorbidities from March 2020 to January 2021 in Afzalipour Hospital in Kerman, Iran. The outcome of patients, hospitalization was recorded as discharge or death. The filtering technique used to score the features and well-known ML algorithms were applied to predict the risk of mortality and LoS of patients. Ensemble Learning methods is also used. To evaluate the performance of the models, different measures including F1, precision, recall, and accuracy were calculated. The TRIPOD guideline assessed transparent reporting.

Results

This study was performed on 1291 patients, including 900 alive and 391 dead patients. Shortness of breath (53.6%), fever (30.1%), and cough (25.3%) were the three most common symptoms in patients. Diabetes mellitus(DM) (31.3%), hypertension (HTN) (27.3%), and ischemic heart disease (IHD) (14.2%) were the three most common chronic comorbidities of patients. Twenty-six important factors were extracted from each patient's record. Gradient boosting model with 84.15% accuracy was the best model for predicting mortality risk and multilayer perceptron (MLP) with rectified linear unit function (MSE = 38.96) was the best model for predicting the LoS. The most common chronic comorbidities among these patients were DM (31.3%), HTN (27.3%), and IHD (14.2%). The most important factors in predicting the risk of mortality were hyperlipidemia, diabetes, asthma, and cancer, and in predicting LoS was shortness of breath.

Conclusion

The results of this study showed that the use of ML algorithms can be a good tool to predict the risk of mortality and LoS of patients with COVID-19 and chronic comorbidities based on physiological conditions, symptoms, and demographic information of patients. The Gradient boosting and MLP algorithms can quickly identify patients at risk of death or long-term hospitalization and notify physicians to do appropriate interventions.

Programmatic Precision Oncology Decision Support for Patients With Gastrointestinal Cancer

PURPOSE

With the growing number of available targeted therapeutics and molecular biomarkers, the optimal care of patients with cancer now depends on a comprehensive understanding of the rapidly evolving landscape of precision oncology, which can be challenging for oncologists to navigate alone.

METHODS

We developed and implemented a precision oncology decision support system, GI TARGET, (Gastrointestinal Treatment Assistance Regarding Genomic Evaluation of Tumors) within the Gastrointestinal Cancer Center at the Dana-Farber Cancer Institute. With a multidisciplinary team, we systematically reviewed tumor molecular profiling for GI tumors and provided molecularly informed clinical recommendations, which included identifying appropriate clinical trials aided by the computational matching platform MatchMiner, suggesting targeted therapy options on or off the US Food and Drug Administration-approved label, and consideration of additional or orthogonal molecular testing.

RESULTS

We reviewed genomic data and provided clinical recommendations for 506 patients with GI cancer who underwent tumor molecular profiling between January and June 2019 and determined follow-up using the electronic health record. Summary reports were provided to 19 medical oncologists for patients with colorectal (n = 198, 39%), pancreatic (n = 124, 24%), esophagogastric (n = 67, 13%), biliary (n = 40, 8%), and other GI cancers. We recommended ≥ 1 precision medicine clinical trial for 80% (406 of 506) of patients, leading to 24 enrollments. We recommended on-label and off-label targeted therapies for 6% (28 of 506) and 25% (125 of 506) of patients, respectively. Recommendations for additional or orthogonal testing were made for 42% (211 of 506) of patients.

CONCLUSION

The integration of precision medicine in routine cancer care through a dedicated multidisciplinary molecular tumor board is scalable and sustainable, and implementation of precision oncology recommendations has clinical utility for patients with cancer.

Applications and perspectives of artificial intelligence, machine learning and “dentronics” in dentistry: A literature review

Objective

The aim of this study was to describe artificial intelligence, machine learning, and "Dentronics" applications and perspectives in dentistry.

Materials and Methods

A literature review was carried out to identify the applications of artificial intelligence in the field of dentistry. A specialized search for information was carried out in three databases such as Scopus, PubMed, and Web of Science. Manuscripts published from January 1988 to November 2021 were analyzed. Articles were included without any restriction by language or country.

Results

Scopus, PubMed, and Web of Science were found to have 215, 1023, and 98 registered manuscripts, respectively. Duplicates (191 manuscripts) were eliminated. Finally, 4 letters, 12 editorials, 5 books, 1 erratum, 54 conference papers, 3 conference reviews, and 222 reviews were excluded.

Conclusions

Artificial intelligence has revolutionized prediction, diagnosis, and therapeutic management in modern dentistry. Finally, artificial intelligence is a potential complement to managing future data in this area.

MLM-based typographical error correction of unstructured medical texts for named entity recognition

Background

Unstructured text in medical records, such as Electronic Health Records, contain an enormous amount of valuable information for research; however, it is difficult to extract and structure important information because of frequent typographical errors. Therefore, improving the quality of data with errors for text analysis is an essential task. To date, few prior studies have been conducted addressing this. Here, we propose a new methodology for extracting important information from unstructured medical texts by overcoming the typographical problem in surgical pathology records related to lung cancer.

Methods

We propose a typo correction model that considers context, based on the Masked Language Model, to solve the problem of typographical errors in real-world medical data. In addition, a word dictionary was used for the typo correction model based on PubMed abstracts. After refining the data through typo correction, fine tuning was performed on pre-trained BERT model. Next, deep learning-based Named Entity Recognition (NER) was performed. By solving the quality problem of medical data, we sought to improve the accuracy of information extraction in unstructured text data.

Results

We compared the performance of the proposed typo correction model based on contextual information with an existing SymSpell model. We confirmed that our proposed model outperformed the existing model in a typographical correction task. The F1-score of the model improved by approximately 5% and 9% when compared with the model without contextual information in the NCBI-disease and surgical pathology record datasets, respectively. In addition, the F1-score of NER after typo correction increased by 2% in the NCBI-disease dataset. There was a significant performance difference of approximately 25% between the before and after typo correction in the Surgical pathology record dataset. This confirmed that typos influenced the information extraction of the unstructured text.

Conclusion

We verified that typographical errors in unstructured text negatively affect the performance of natural language processing tasks. The proposed method of a typo correction model outperformed the existing SymSpell model. This study shows that the proposed model is robust and can be applied in real-world environments by focusing on the typos that cause difficulties in analyzing unstructured medical text.

Improving infectious adverse event reporting for children and adolescents enrolled in clinical trials for acute lymphoblastic leukemia: A report from the Children's Oncology Group

Infections cause substantial morbidity for children with acute lymphoblastic leukemia (ALL). Therefore, accurate characterization of infectious adverse events (AEs) reported on clinical trials is imperative to defining, comparing, and managing safety and toxicity. Here, we describe key processes implemented to improve reporting of infectious AEs on two active phase III Children's Oncology Group (COG) ALL trials. Processes include: (a) identifying infections as a targeted toxicity, (b) incorporation of infection-specific case report form questions, and (c) physician review of AEs with real-time data cleaning. Preliminary assessment of these processes suggests improved reporting, as well as opportunities for further improvement.

The Challenges of Regulating Artificial Intelligence in Healthcare Comment on "Clinical Decision Support and New Regulatory Frameworks for Medical Devices: Are We Ready for It? - A Viewpoint Paper"

Regulation of health technologies must be rigorous, instilling trust among both healthcare providers and patients. This is especially important for the control and supervision of the growing use of artificial intelligence in healthcare. In this commentary on the accompanying piece by Van Laere and colleagues, we set out the scope for applying artificial intelligence in the healthcare sector and outline five key challenges that regulators face in dealing with these modern-day technologies. Addressing these challenges will not be easy. While artificial intelligence applications in healthcare have already made rapid progress and benefitted patients, these applications clearly hold even more potential for future developments. Yet it is vital that the regulatory environment keep up with this fast-evolving space of healthcare in order to anticipate and, to the extent possible, prevent the risks that may arise.

Natural Language Processing of Computed Tomography Reports to Label Metastatic Phenotypes With Prognostic Significance in Patients With Colorectal Cancer

PURPOSE

Natural language processing (NLP) applied to radiology reports can help identify clinically relevant M1 subcategories of patients with colorectal cancer (CRC). The primary purpose was to compare the overall survival (OS) of CRC according to American Joint Committee on Cancer TNM staging and explore an alternative classification. The secondary objective was to estimate the frequency of metastasis for each organ.

METHODS

Retrospective study of CRC who underwent computed tomography (CT) chest, abdomen, and pelvis between July 1, 2009, and March 26, 2019, at a tertiary cancer center, previously labeled for the presence or absence of metastasis by an NLP prediction model. Patients were classified in M0, M1a, M1b, and M1c (American Joint Committee on Cancer), or an alternative classification on the basis of the metastasis organ number: M1, single; M2, two; M3, three or more organs. Cox regression models were used to estimate hazard ratios; Kaplan-Meier curves were used to visualize survival curves using the two M1 subclassifications.

RESULTS

Nine thousand nine hundred twenty-eight patients with a total of 48,408 CT chest, abdomen, and pelvis reports were included. On the basis of NLP prediction, the median OS of M1a, M1b, and M1c was 4.47, 1.72, and 1.52 years, respectively. The median OS of M1, M2, and M3 was 4.24, 2.05, and 1.04 years, respectively. Metastases occurred most often in liver (35.8%), abdominopelvic lymph nodes (32.9%), lungs (29.3%), peritoneum (22.0%), thoracic nodes (19.9%), bones (9.2%), and pelvic organs (7.5%). Spleen and adrenal metastases occurred in < 5%.

CONCLUSION

NLP applied to a large radiology report database can identify clinically relevant metastatic phenotypes and be used to investigate new M1 substaging for CRC. Patients with three or more metastatic disease organs have the worst prognosis, with an OS of 1 year.

Deep Learning-based Assessment of Oncologic Outcomes from Natural Language Processing of Structured Radiology Reports

PURPOSE

To train a deep natural language processing (NLP) model, using data mined structured oncology reports (SOR), for rapid tumor response category (TRC) classification from free-text oncology reports (FTOR) and to compare its performance with human readers and conventional NLP algorithms.

MATERIALS AND METHODS

In this retrospective study, databases of three independent radiology departments were queried for SOR and FTOR dated from March 2018 to August 2021. An automated data mining and curation pipeline was developed to extract Response Evaluation Criteria in Solid Tumors-related TRCs for SOR for ground truth definition. The deep NLP bidirectional encoder representations from transformers (BERT) model and three feature-rich algorithms were trained on SOR to predict TRCs in FTOR. Models' F1 scores were compared against scores of radiologists, medical students, and radiology technologist students. Lexical and semantic analyses were conducted to investigate human and model performance on FTOR.

RESULTS

Oncologic findings and TRCs were accurately mined from 9653 of 12 833 (75.2%) queried SOR, yielding oncology reports from 10 455 patients (mean age, 60 years ± 14 [SD]; 5303 women) who met inclusion criteria. On 802 FTOR in the test set, BERT achieved better TRC classification results (F1, 0.70; 95% CI: 0.68, 0.73) than the best-performing reference linear support vector classifier (F1, 0.63; 95% CI: 0.61, 0.66) and technologist students (F1, 0.65; 95% CI: 0.63, 0.67), had similar performance to medical students (F1, 0.73; 95% CI: 0.72, 0.75), but was inferior to radiologists (F1, 0.79; 95% CI: 0.78, 0.81). Lexical complexity and semantic ambiguities in FTOR influenced human and model performance, revealing maximum F1 score drops of -0.17 and -0.19, respectively.

CONCLUSION

The developed deep NLP model reached the performance level of medical students but not radiologists in curating oncologic outcomes from radiology FTOR.Keywords: Neural Networks, Computer Applications-Detection/Diagnosis, Oncology, Research Design, Staging, Tumor Response, Comparative Studies, Decision Analysis, Experimental Investigations, Observer Performance, Outcomes Analysis Supplemental material is available for this article. © RSNA, 2022.

Natural Language Processing in Radiology: Update on Clinical Applications

Radiological reports are a valuable source of information used to guide clinical care and support research. Organizing and managing this content, however, frequently requires several manual curations due to the more common unstructured nature of the reports. However, manual review of these reports for clinical knowledge extraction is costly and time-consuming. Natural language processing (NLP) is a set of methods developed to extract structured meaning from a body of text and can be used to optimize the workflow of health care professionals. Specifically, NLP methods can help radiologists as decision support systems and improve the management of patients' medical data. In this study, we highlight the opportunities offered by NLP in the field of radiology. A comprehensive review of the most commonly used NLP methods to extract information from radiological reports and the development of tools to improve radiological workflow using this information is presented. Finally, we review the important limitations of these tools and discuss the relevant observations and trends in the application of NLP to radiology that could benefit the field in the future.

Development and validation of an abnormality-derived deep-learning diagnostic system for major respiratory diseases

Respiratory diseases impose a tremendous global health burden on large patient populations. In this study, we aimed to develop DeepMRD^TR, a deep learning-based medical image interpretation system for the diagnosis of major respiratory diseases based on the automated identification of a wide range of radiological abnormalities through computed tomography (CT) and chest X-ray (CXR) from real-world, large-scale datasets. DeepMRD^TR comprises four networks (two CT-Nets and two CXR-Nets) that exploit contrastive learning to generate pre-training parameters that are fine-tuned on the retrospective dataset collected from a single institution. The performance of DeepMRD^TR was evaluated for abnormality identification and disease diagnosis on data from two different institutions: one was an internal testing dataset from the same institution as the training data and the second was collected from an external institution to evaluate the model generalizability and robustness to an unrelated population dataset. In such a difficult multi-class diagnosis task, our system achieved the average area under the receiver operating characteristic curve (AUC) of 0.856 (95% confidence interval (CI):0.843–0.868) and 0.841 (95%CI:0.832–0.887) for abnormality identification, and 0.900 (95%CI:0.872–0.958) and 0.866 (95%CI:0.832–0.887) for major respiratory diseases’ diagnosis on CT and CXR datasets, respectively. Furthermore, to achieve a clinically actionable diagnosis, we deployed a preliminary version of DeepMRD^TR into the clinical workflow, which was performed on par with senior experts in disease diagnosis, with an AUC of 0.890 and a Cohen’s k of 0.746–0.877 at a reasonable timescale; these findings demonstrate the potential to accelerate the medical workflow to facilitate early diagnosis as a triage tool for respiratory diseases which supports improved clinical diagnoses and decision-making.

An Artificial Intelligence-Based Tool for Data Analysis and Prognosis in Cancer Patients: Results from the Clarify Study

Simple Summary

Cancer is associated with significant morbimortality worldwide. Although significant advances have been made in the last few decades in terms of early detection and treatment, providing personalized care remains a challenge. Artificial intelligence (AI) has emerged as a means of improving cancer care with the use of computer science. Identification of risk factors for poor prognosis and patient profiling with AI techniques and tools is feasible and has potential application in clinical settings, including surveillance management. The goal of this study is to present an AI-based solution tool for cancer patients data analysis and improve their management by identifying clinical factors associated with relapse and survival, developing a prognostic model that identifies features associated with poor prognosis, and stratifying patients by risk.

Abstract

Background: Artificial intelligence (AI) has contributed substantially in recent years to the resolution of different biomedical problems, including cancer. However, AI tools with significant and widespread impact in oncology remain scarce. The goal of this study is to present an AI-based solution tool for cancer patients data analysis that assists clinicians in identifying the clinical factors associated with poor prognosis, relapse and survival, and to develop a prognostic model that stratifies patients by risk. Materials and Methods: We used clinical data from 5275 patients diagnosed with non-small cell lung cancer, breast cancer, and non-Hodgkin lymphoma at Hospital Universitario Puerta de Hierro-Majadahonda. Accessible clinical parameters measured with a wearable device and quality of life questionnaires data were also collected. Results: Using an AI-tool, data from 5275 cancer patients were analyzed, integrating clinical data, questionnaires data, and data collected from wearable devices. Descriptive analyses were performed in order to explore the patients’ characteristics, survival probabilities were calculated, and a prognostic model identified low and high-risk profile patients. Conclusion: Overall, the reconstruction of the population’s risk profile for the cancer-specific predictive model was achieved and proved useful in clinical practice using artificial intelligence. It has potential application in clinical settings to improve risk stratification, early detection, and surveillance management of cancer patients.

Deep Learning for Cancer Symptoms Monitoring on the Basis of Electronic Health Record Unstructured Clinical Notes

PURPOSE

Symptoms are vital outcomes for cancer clinical trials, observational research, and population-level surveillance. Patient-reported outcomes (PROs) are valuable for monitoring symptoms, yet there are many challenges to collecting PROs at scale. We sought to develop, test, and externally validate a deep learning model to extract symptoms from unstructured clinical notes in the electronic health record.

METHODS

We randomly selected 1,225 outpatient progress notes from among patients treated at the Dana-Farber Cancer Institute between January 2016 and December 2019 and used 1,125 notes as our training/validation data set and 100 notes as our test data set. We evaluated the performance of 10 deep learning models for detecting 80 symptoms included in the National Cancer Institute's Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) framework. Model performance as compared with manual chart abstraction was assessed using standard metrics, and the highest performer was externally validated on a sample of 100 physician notes from a different clinical context.

RESULTS

In our training and test data sets, 75 of the 80 candidate symptoms were identified. The ELECTRA-small model had the highest performance for symptom identification at the token level (ie, at the individual symptom level), with an F1 of 0.87 and a processing time of 3.95 seconds per note. For the 10 most common symptoms in the test data set, the F1 score ranged from 0.98 for anxious to 0.86 for fatigue. For external validation of the same symptoms, the note-level performance ranged from F1 = 0.97 for diarrhea and dizziness to F1 = 0.73 for swelling.

CONCLUSION

Training a deep learning model to identify a wide range of electronic health record-documented symptoms relevant to cancer care is feasible. This approach could be used at the health system scale to complement to electronic PROs.

Concordance of Clinician-Documented and Imaging Response in Patients With Stage IV Non-Small Cell Lung Cancer Treated With First-Line Therapy

IMPORTANCE

In observational oncology studies of solid tumors, response to treatment can be evaluated based on electronic health record (EHR) documentation (clinician-assessed response [CAR]), an approach different from standardized radiologist-measured response (Response Evaluation Criteria in Solid Tumours [RECIST] 1.1).

OBJECTIVE

To evaluate the feasibility of an imaging response based on RECIST (IRb-RECIST) and the concordance between CAR and imaging response based on RECIST assessments, and investigate discordance causes.

DESIGN, SETTING, AND PARTICIPANTS

This cohort study used an EHR-derived, deidentified database that included patients with stage IV non-small cell lung cancer (NSCLC) diagnosed between January 1, 2011, to June 30, 2019, selected from 3 study sites. Data analysis was conducted in August, 2020.

EXPOSURES

Undergoing first-line therapy and imaging assessments of response to treatment.

MAIN OUTCOMES AND MEASURES

In this study, CAR assessments (referred to in prior publications as "real-world response" [rwR]) were defined as clinician-documented changes in disease burden at radiologic evaluation time points; they were abstracted manually and assigned to response categories. The RECIST-based assessments accommodated routine practice patterns by using a modified version of RECIST 1.1 (IRb-RECIST), with independent radiology reads. Concordance was calculated as the percent agreement across all response categories and across a dichotomous stratification (response [complete or partial] vs no response), unconfirmed or confirmed.

RESULTS

This study found that, in 100 patients evaluated for concordance, agreement between CAR and IRb-RECIST was 71% (95% CI, 61%-80%), and 74% (95% CI, 64%-82%) for confirmed and unconfirmed response, respectively. There were more responders using CAR than IRb-RECIST (40 vs 29 with confirmation; 64 vs 43 without confirmation). The main sources of discordance were the different use of thresholds for tumor size changes by RECIST vs routine care, and unavailable baseline or follow-up scans resulting in inconsistent anatomic coverage over time.

CONCLUSIONS AND RELEVANCE

In this cohort study of patients with stage IV NSCLC, we collected routine-care imaging, showing the feasibility of response evaluation using IRb-RECIST criteria with independent centralized review. Concordance between CAR and centralized IRb-RECIST was moderate. Future work is needed to evaluate the generalizability of these results to broader populations, and investigate concordance in other clinical settings.

Automated Radiology-Arthroscopy Correlation of Knee Meniscal Tears Using Natural Language Processing Algorithms

RATIONALE AND OBJECTIVES

Train and apply natural language processing (NLP) algorithms for automated radiology-arthroscopy correlation of meniscal tears.

MATERIALS AND METHODS

In this retrospective single-institution study, we trained supervised machine learning models (logistic regression, support vector machine, and random forest) to detect medial or lateral meniscus tears on free-text MRI reports. We trained and evaluated model performances with cross-validation using 3593 manually annotated knee MRI reports. To assess radiology-arthroscopy correlation, we then randomly partitioned this dataset 80:20 for training and testing, where 108 test set MRIs were followed by knee arthroscopy within 1 year. These free-text arthroscopy reports were also manually annotated. The NLP algorithms trained on the knee MRI training dataset were then evaluated on the MRI and arthroscopy report test datasets. We assessed radiology-arthroscopy agreement using the ensembled NLP-extracted findings versus manually annotated findings.

RESULTS

The NLP models showed high cross-validation performance for meniscal tear detection on knee MRI reports (medial meniscus F1 scores 0.93-0.94, lateral meniscus F1 scores 0.86-0.88). When these algorithms were evaluated on arthroscopy reports, despite never training on arthroscopy reports, performance was similar, though higher with model ensembling (medial meniscus F1 score 0.97, lateral meniscus F1 score 0.99). However, ensembling did not improve performance on knee MRI reports. In the radiology-arthroscopy test set, the ensembled NLP models were able to detect mismatches between MRI and arthroscopy reports with sensitivity 79% and specificity 87%.

CONCLUSION

Radiology-arthroscopy correlation can be automated for knee meniscal tears using NLP algorithms, which shows promise for education and quality improvement.

Developing a Cancer Digital Twin: Supervised Metastases Detection From Consecutive Structured Radiology Reports

The development of digital cancer twins relies on the capture of high-resolution representations of individual cancer patients throughout the course of their treatment. Our research aims to improve the detection of metastatic disease over time from structured radiology reports by exposing prediction models to historical information. We demonstrate that Natural language processing (NLP) can generate better weak labels for semi-supervised classification of computed tomography (CT) reports when it is exposed to consecutive reports through a patient's treatment history. Around 714,454 structured radiology reports from Memorial Sloan Kettering Cancer Center adhering to a standardized departmental structured template were used for model development with a subset of the reports included for validation. To develop the models, a subset of the reports was curated for ground-truth: 7,732 total reports in the lung metastases dataset from 867 individual patients; 2,777 reports in the liver metastases dataset from 315 patients; and 4,107 reports in the adrenal metastases dataset from 404 patients. We use NLP to extract and encode important features from the structured text reports, which are then used to develop, train, and validate models. Three models—a simple convolutional neural network (CNN), a CNN augmented with an attention layer, and a recurrent neural network (RNN)—were developed to classify the type of metastatic disease and validated against the ground truth labels. The models use features from consecutive structured text radiology reports of a patient to predict the presence of metastatic disease in the reports. A single-report model, previously developed to analyze one report instead of multiple past reports, is included and the results from all four models are compared based on accuracy, precision, recall, and F1-score. The best model is used to label all 714,454 reports to generate metastases maps. Our results suggest that NLP models can extract cancer progression patterns from multiple consecutive reports and predict the presence of metastatic disease in multiple organs with higher performance when compared with a single-report-based prediction. It demonstrates a promising automated approach to label large numbers of radiology reports without involving human experts in a time- and cost-effective manner and enables tracking of cancer progression over time.

A Scalable Quality Assurance Process for Curating Oncology Electronic Health Records: The Project GENIE Biopharma Collaborative Approach

The American Association for Cancer Research Project Genomics Evidence Neoplasia Information Exchange Biopharma Collaborative is a multi-institution effort to build a pan-cancer repository of genomic and clinical data curated from the electronic health record. For the research community to be confident that data extracted from electronic health record text are reliable, transparency of the approach used to ensure data quality is essential.

Transparent QA processes for GENIE BPC ensure that the data can be used to support advances in precision oncology OR @jessicalavs of @MSKBiostats & coauthors discuss @AACR Project GENIE BPC, a multi-institution effort to aggregate clinical plus genomic data for patients with cancer. Transparent QA processes for GENIE BPC ensure that the data can be used to support advances in precision oncology.

Harnessing multimodal data integration to advance precision oncology

Advances in quantitative biomarker development have accelerated new forms of data-driven insights for patients with cancer. However, most approaches are limited to a single mode of data, leaving integrated approaches across modalities relatively underdeveloped. Multimodal integration of advanced molecular diagnostics, radiological and histological imaging, and codified clinical data presents opportunities to advance precision oncology beyond genomics and standard molecular techniques. However, most medical datasets are still too sparse to be useful for the training of modern machine learning techniques, and significant challenges remain before this is remedied. Combined efforts of data engineering, computational methods for analysis of heterogeneous data and instantiation of synergistic data models in biomedical research are required for success. In this Perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods. Advancing along this direction will result in a reimagined class of multimodal biomarkers to propel the field of precision oncology in the coming decade.