Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms

Differences between human and machine perception in medical diagnosis

Deep neural networks (DNNs) show promise in image-based medical diagnosis, but cannot be fully trusted since they can fail for reasons unrelated to underlying pathology. Humans are less likely to make such superficial mistakes, since they use features that are grounded on medical science. It is therefore important to know whether DNNs use different features than humans. Towards this end, we propose a framework for comparing human and machine perception in medical diagnosis. We frame the comparison in terms of perturbation robustness, and mitigate Simpson's paradox by performing a subgroup analysis. The framework is demonstrated with a case study in breast cancer screening, where we separately analyze microcalcifications and soft tissue lesions. While it is inconclusive whether humans and DNNs use different features to detect microcalcifications, we find that for soft tissue lesions, DNNs rely on high frequency components ignored by radiologists. Moreover, these features are located outside of the region of the images found most suspicious by radiologists. This difference between humans and machines was only visible through subgroup analysis, which highlights the importance of incorporating medical domain knowledge into the comparison.

Artificial Intelligence in Emergency Medicine: Benefits, Risks, and Recommendations

BACKGROUND

Artificial intelligence (AI) can be described as the use of computers to perform tasks that formerly required human cognition. The American Medical Association prefers the term 'augmented intelligence' over 'artificial intelligence' to emphasize the assistive role of computers in enhancing physician skills as opposed to replacing them. The integration of AI into emergency medicine, and clinical practice at large, has increased in recent years, and that trend is likely to continue.

DISCUSSION

AI has demonstrated substantial potential benefit for physicians and patients. These benefits are transforming the therapeutic relationship from the traditional physician-patient dyad into a triadic doctor-patient-machine relationship. New AI technologies, however, require careful vetting, legal standards, patient safeguards, and provider education. Emergency physicians (EPs) should recognize the limits and risks of AI as well as its potential benefits.

CONCLUSIONS

EPs must learn to partner with, not capitulate to, AI. AI has proven to be superior to, or on a par with, certain physician skills, such as interpreting radiographs and making diagnoses based on visual cues, such as skin cancer. AI can provide cognitive assistance, but EPs must interpret AI results within the clinical context of individual patients. They must also advocate for patient confidentiality, professional liability coverage, and the essential role of specialty-trained EPs.

Multi-Task Fusion for Improving Mammography Screening Data Classification

Machine learning and deep learning methods have become essential for computer-assisted prediction in medicine, with a growing number of applications also in the field of mammography. Typically these algorithms are trained for a specific task, e.g., the classification of lesions or the prediction of a mammogram's pathology status. To obtain a comprehensive view of a patient, models which were all trained for the same task(s) are subsequently ensembled or combined. In this work, we propose a pipeline approach, where we first train a set of individual, task-specific models and subsequently investigate the fusion thereof, which is in contrast to the standard model ensembling strategy. We fuse model predictions and high-level features from deep learning models with hybrid patient models to build stronger predictors on patient level. To this end, we propose a multi-branch deep learning model which efficiently fuses features across different tasks and mammograms to obtain a comprehensive patient-level prediction. We train and evaluate our full pipeline on public mammography data, i.e., DDSM and its curated version CBIS-DDSM, and report an AUC score of 0.962 for predicting the presence of any lesion and 0.791 for predicting the presence of malignant lesions on patient level. Overall, our fusion approaches improve AUC scores significantly by up to 0.04 compared to standard model ensembling. Moreover, by providing not only global patient-level predictions but also task-specific model results that are related to radiological features, our pipeline aims to closely support the reading workflow of radiologists.

An overview of artificial intelligence in oncology

Cancer is associated with significant morbimortality globally. Advances in screening, diagnosis, management and survivorship were substantial in the last decades, however, challenges in providing personalized and data-oriented care remain. Artificial intelligence (AI), a branch of computer science used for predictions and automation, has emerged as potential solution to improve the healthcare journey and to promote precision in healthcare. AI applications in oncology include, but are not limited to, optimization of cancer research, improvement of clinical practice (eg., prediction of the association of multiple parameters and outcomes - prognosis and response) and better understanding of tumor molecular biology. In this review, we examine the current state of AI in oncology, including fundamentals, current applications, limitations and future perspectives.

Innovation en santé au service des urgences et des cliniciens-chercheurs : le parcours et l’évaluation clinique d’un dispositif médical

L’alliance du monde de la santé avec celui des nouvelles technologies peut générer un nouveau champ pour la recherche et l’exercice médical. La nouvelle réglementation européenne du marquage Communauté européenne (CE) (2017/745) impose aujourd’hui à tout dispositif médical (DM), y compris les logiciels informatiques, une évaluation clinique rigoureuse. Les services d’urgence sont particulièrement à la pointe en matière d’appétence pour les nouvelles technologies. Ils sont de plus en plus sollicités par des entreprises existantes ou en création, travaillant dans le champ de l’ingénierie à destination du domaine de la santé, mais également par des chercheurs en ingénierie fondamentale. Au regard de ces éléments, il nous semblait pertinent pour les médecins et cliniciens-chercheurs en médecine d’urgence de définir les étapes et jalons d’un projet innovant en santé comprenant un DM. Après avoir passé les premiers stades de développement, matérialisables par l’échelle Technology Readiness Level (TRL), des investigations cliniques sont nécessaires afin d’obtenir un marquage CE auprès d’un organisme notifié. Selon les caractéristiques du DM, différentes modalités de prise en charge par la collectivité sont possibles et peuvent nécessiter des preuves cliniques, médicoéconomiques ou organisationnelles supplémentaires. Les attentes des acteurs et étapes associées au processus d’accès au marché peuvent être un frein au développement des futurs DM. L’association entre chercheurs, cliniciens-chercheurs et industriels devrait permettre l’émergence d’outils novateurs et pratiques pour la médecine d’urgence de demain, à condition d’avoir une bonne connaissance de ces étapes.

The Role of Artificial Intelligence in Early Cancer Diagnosis

Improving the proportion of patients diagnosed with early-stage cancer is a key priority of the World Health Organisation. In many tumour groups, screening programmes have led to improvements in survival, but patient selection and risk stratification are key challenges. In addition, there are concerns about limited diagnostic workforces, particularly in light of the COVID-19 pandemic, placing a strain on pathology and radiology services. In this review, we discuss how artificial intelligence algorithms could assist clinicians in (1) screening asymptomatic patients at risk of cancer, (2) investigating and triaging symptomatic patients, and (3) more effectively diagnosing cancer recurrence. We provide an overview of the main artificial intelligence approaches, including historical models such as logistic regression, as well as deep learning and neural networks, and highlight their early diagnosis applications. Many data types are suitable for computational analysis, including electronic healthcare records, diagnostic images, pathology slides and peripheral blood, and we provide examples of how these data can be utilised to diagnose cancer. We also discuss the potential clinical implications for artificial intelligence algorithms, including an overview of models currently used in clinical practice. Finally, we discuss the potential limitations and pitfalls, including ethical concerns, resource demands, data security and reporting standards.

Advancements in Oncology with Artificial Intelligence—A Review Article

Simple Summary

With the advancement of artificial intelligence, including machine learning, the field of oncology has seen promising results in cancer detection and classification, epigenetics, drug discovery, and prognostication. In this review, we describe what artificial intelligence is and its function, as well as comprehensively summarize its evolution and role in breast, colorectal, and central nervous system cancers. Understanding the origin and current accomplishments might be essential to improve the quality, accuracy, generalizability, cost-effectiveness, and reliability of artificial intelligence models that can be used in worldwide clinical practice. Students and researchers in the medical field will benefit from a deeper understanding of how to use integrative AI in oncology for innovation and research.

Abstract

Well-trained machine learning (ML) and artificial intelligence (AI) systems can provide clinicians with therapeutic assistance, potentially increasing efficiency and improving efficacy. ML has demonstrated high accuracy in oncology-related diagnostic imaging, including screening mammography interpretation, colon polyp detection, glioma classification, and grading. By utilizing ML techniques, the manual steps of detecting and segmenting lesions are greatly reduced. ML-based tumor imaging analysis is independent of the experience level of evaluating physicians, and the results are expected to be more standardized and accurate. One of the biggest challenges is its generalizability worldwide. The current detection and screening methods for colon polyps and breast cancer have a vast amount of data, so they are ideal areas for studying the global standardization of artificial intelligence. Central nervous system cancers are rare and have poor prognoses based on current management standards. ML offers the prospect of unraveling undiscovered features from routinely acquired neuroimaging for improving treatment planning, prognostication, monitoring, and response assessment of CNS tumors such as gliomas. By studying AI in such rare cancer types, standard management methods may be improved by augmenting personalized/precision medicine. This review aims to provide clinicians and medical researchers with a basic understanding of how ML works and its role in oncology, especially in breast cancer, colorectal cancer, and primary and metastatic brain cancer. Understanding AI basics, current achievements, and future challenges are crucial in advancing the use of AI in oncology.

Reader Perceptions and Impact of AI on CT Assessment of Air Trapping

Quantitative imaging measurements can be facilitated by artificial intelligence (AI) algorithms, but how they might impact decision-making and be perceived by radiologists remains uncertain. After creation of a dedicated inspiratory-expiratory CT examination and concurrent deployment of a quantitative AI algorithm for assessing air trapping, five cardiothoracic radiologists retrospectively evaluated severity of air trapping on 17 examination studies. Air trapping severity of each lobe was evaluated in three stages: qualitatively (visually); semiquantitatively, allowing manual region-of-interest measurements; and quantitatively, using results from an AI algorithm. Readers were surveyed on each case for their perceptions of the AI algorithm. The algorithm improved interreader agreement (intraclass correlation coefficients: visual, 0.28; semiquantitative, 0.40; quantitative, 0.84; P < .001) and improved correlation with pulmonary function testing (forced expiratory volume in 1 second-to-forced vital capacity ratio) (visual r = -0.26, semiquantitative r = -0.32, quantitative r = -0.44). Readers perceived moderate agreement with the AI algorithm (Likert scale average, 3.7 of 5), a mild impact on their final assessment (average, 2.6), and a neutral perception of overall utility (average, 3.5). Though the AI algorithm objectively improved interreader consistency and correlation with pulmonary function testing, individual readers did not immediately perceive this benefit, revealing a potential barrier to clinical adoption. Keywords: Technology Assessment, Quantification © RSNA, 2021.

Deep learning in image-based breast and cervical cancer detection: a systematic review and meta-analysis

Accurate early detection of breast and cervical cancer is vital for treatment success. Here, we conduct a meta-analysis to assess the diagnostic performance of deep learning (DL) algorithms for early breast and cervical cancer identification. Four subgroups are also investigated: cancer type (breast or cervical), validation type (internal or external), imaging modalities (mammography, ultrasound, cytology, or colposcopy), and DL algorithms versus clinicians. Thirty-five studies are deemed eligible for systematic review, 20 of which are meta-analyzed, with a pooled sensitivity of 88% (95% CI 85–90%), specificity of 84% (79–87%), and AUC of 0.92 (0.90–0.94). Acceptable diagnostic performance with analogous DL algorithms was highlighted across all subgroups. Therefore, DL algorithms could be useful for detecting breast and cervical cancer using medical imaging, having equivalent performance to human clinicians. However, this tentative assertion is based on studies with relatively poor designs and reporting, which likely caused bias and overestimated algorithm performance. Evidence-based, standardized guidelines around study methods and reporting are required to improve the quality of DL research.

Improved automated early detection of breast cancer based on high resolution 3D micro-CT microcalcification images

Background

The detection of suspicious microcalcifications on mammography represents one of the earliest signs of a malignant breast tumor. Assessing microcalcifications’ characteristics based on their appearance on 2D breast imaging modalities is in many cases challenging for radiologists. The aims of this study were to: (a) analyse the association of shape and texture properties of breast microcalcifications (extracted by scanning breast tissue with a high resolution 3D scanner) with malignancy, (b) evaluate microcalcifications’ potential to diagnose benign/malignant patients.

Methods

Biopsy samples of 94 female patients with suspicious microcalcifications detected during a mammography, were scanned using a micro-CT scanner at a resolution of 9 μm. Several preprocessing techniques were applied on 3504 extracted microcalcifications. A high amount of radiomic features were extracted in an attempt to capture differences among microcalcifications occurring in benign and malignant lesions. Machine learning algorithms were used to diagnose: (a) individual microcalcifications, (b) samples. For the samples, several methodologies to combine individual microcalcification results into sample results were evaluated.

Results

We could classify individual microcalcifications with 77.32% accuracy, 61.15% sensitivity and 89.76% specificity. At the sample level diagnosis, we achieved an accuracy of 84.04%, sensitivity of 86.27% and specificity of 81.39%.

Conclusions

By studying microcalcifications’ characteristics at a level of details beyond what is currently possible by using conventional breast imaging modalities, our classification results demonstrated a strong association between breast microcalcifications and malignancies. Microcalcification’s texture features extracted in transform domains, have higher discriminating power to classify benign/malignant individual microcalcifications and samples compared to pure shape-features.

Independent External Validation of Artificial Intelligence Algorithms for Automated Interpretation of Screening Mammography: A Systematic Review

PURPOSE

The aim of this study was to describe the current state of science regarding independent external validation of artificial intelligence (AI) technologies for screening mammography.

METHODS

A systematic review was performed across five databases (Embase, PubMed, IEEE Explore, Engineer Village, and arXiv) through December 10, 2020. Studies that used screening examinations from real-world settings to externally validate AI algorithms for mammographic cancer detection were included. The main outcome was diagnostic accuracy, defined by area under the receiver operating characteristic curve (AUC). Performance was also compared between radiologists and either stand-alone AI or combined radiologist and AI interpretation. Study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies 2 tool.

RESULTS

After data extraction, 13 studies met the inclusion criteria (148,361 total patients). Most studies (77% [n = 10]) evaluated commercially available AI algorithms. Studies included retrospective reader studies (46% [n = 6]), retrospective simulation studies (38% [n = 5]), or both (15% [n = 2]). Across 5 studies comparing stand-alone AI with radiologists, 60% (n = 3) demonstrated improved accuracy with AI (AUC improvement range, 0.02-0.13). All 5 studies comparing combined radiologist and AI interpretation with radiologists alone demonstrated improved accuracy with AI (AUC improvement range, 0.028-0.115). Most studies had risk for bias or applicability concerns for patient selection (69% [n = 9]) and the reference standard (69% [n = 9]). Only two studies obtained ground-truth cancer outcomes through regional cancer registry linkage.

CONCLUSIONS

To date, external validation efforts for AI screening mammographic technologies suggest small potential diagnostic accuracy improvements but have been retrospective in nature and suffer from risk for bias and applicability concerns.

Online or Offline: Does It Matter? A Review of Existing Interpretation Approaches and Their Effect on Screening Mammography Metrics, Patient Satisfaction, and Cost

The ideal practice routine for screening mammography would optimize performance metrics and minimize costs, while also maximizing patient satisfaction. The main approaches to screening mammography interpretation include batch offline, non-batch offline, interrupted online, and uninterrupted online reading, each of which has its own advantages and drawbacks. This article reviews the current literature on approaches to screening mammography interpretation, potential effects of newer technologies, and promising artificial intelligence resources that could improve workflow efficiency in the future.

An artificial intelligence-assisted diagnostic system improves the accuracy of image diagnosis of uterine cervical lesions

The present study created an artificial intelligence (AI)-automated diagnostics system for uterine cervical lesions and assessed the performance of these images for AI diagnostic imaging of pathological cervical lesions. A total of 463 colposcopic images were analyzed. The traditional colposcopy diagnoses were compared to those obtained by AI image diagnosis. Next, 100 images were presented to a panel of 32 gynecologists who independently examined each image in a blinded fashion and diagnosed them for four categories of tumors. Then, the 32 gynecologists revisited their diagnosis for each image after being informed of the AI diagnosis. The present study assessed any changes in physician diagnosis and the accuracy of AI-image-assisted diagnosis (AISD). The accuracy of AI was 57.8% for normal, 35.4% for cervical intraepithelial neoplasia (CIN)1, 40.5% for CIN2-3 and 44.2% for invasive cancer. The accuracy of gynecologist diagnoses from cervical pathological images, before knowing the AI image diagnosis, was 54.4% for CIN2-3 and 38.9% for invasive cancer. After learning of the AISD, their accuracy improved to 58.0% for CIN2-3 and 48.5% for invasive cancer. AI-assisted image diagnosis was able to improve gynecologist diagnosis accuracy significantly (P<0.01) for invasive cancer and tended to improve their accuracy for CIN2-3 (P=0.14).

Machine Learning for Workflow Applications in Screening Mammography: Systematic Review and Meta-Analysis

Background Advances in computer processing and improvements in data availability have led to the development of machine learning (ML) techniques for mammographic imaging. Purpose To evaluate the reported performance of stand-alone ML applications for screening mammography workflow. Materials and Methods Ovid Embase, Ovid Medline, Cochrane Central Register of Controlled Trials, Scopus, and Web of Science literature databases were searched for relevant studies published from January 2012 to September 2020. The study was registered with the PROSPERO International Prospective Register of Systematic Reviews (protocol no. CRD42019156016). Stand-alone technology was defined as a ML algorithm that can be used independently of a human reader. Studies were quality assessed using the Quality Assessment of Diagnostic Accuracy Studies 2 and the Prediction Model Risk of Bias Assessment Tool, and reporting was evaluated using the Checklist for Artificial Intelligence in Medical Imaging. A primary meta-analysis included the top-performing algorithm and corresponding reader performance from which pooled summary estimates for the area under the receiver operating characteristic curve (AUC) were calculated using a bivariate model. Results Fourteen articles were included, which detailed 15 studies for stand-alone detection (n = 8) and triage (n = 7). Triage studies reported that 17%-91% of normal mammograms identified could be read by adapted screening, while "missing" an estimated 0%-7% of cancers. In total, an estimated 185 252 cases from three countries with more than 39 readers were included in the primary meta-analysis. The pooled sensitivity, specificity, and AUC was 75.4% (95% CI: 65.6, 83.2; P = .11), 90.6% (95% CI: 82.9, 95.0; P = .40), and 0.89 (95% CI: 0.84, 0.98), respectively, for algorithms, and 73.0% (95% CI: 60.7, 82.6), 88.6% (95% CI: 72.4, 95.8), and 0.85 (95% CI: 0.78, 0.97), respectively, for readers. Conclusion Machine learning (ML) algorithms that demonstrate a stand-alone application in mammographic screening workflows achieve or even exceed human reader detection performance and improve efficiency. However, this evidence is from a small number of retrospective studies. Therefore, further rigorous independent external prospective testing of ML algorithms to assess performance at preassigned thresholds is required to support these claims. ©RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Whitman and Moseley in this issue.

Artificial intelligence and machine learning in emergency medicine: a narrative review

AIM

The emergence and evolution of artificial intelligence (AI) has generated increasing interest in machine learning applications for health care. Specifically, researchers are grasping the potential of machine learning solutions to enhance the quality of care in emergency medicine.

METHODS

We undertook a narrative review of published works on machine learning applications in emergency medicine and provide a synopsis of recent developments.

RESULTS

This review describes fundamental concepts of machine learning and presents clinical applications for triage, risk stratification specific to disease, medical imaging, and emergency department operations. Additionally, we consider how machine learning models could contribute to the improvement of causal inference in medicine, and to conclude, we discuss barriers to safe implementation of AI.

CONCLUSION

We intend that this review serves as an introduction to AI and machine learning in emergency medicine.

Improving the Performance of Radiologists Using Artificial Intelligence-Based Detection Support Software for Mammography: A Multi-Reader Study

Objective

To evaluate whether artificial intelligence (AI) for detecting breast cancer on mammography can improve the performance and time efficiency of radiologists reading mammograms.

Materials and Methods

A commercial deep learning-based software for mammography was validated using external data collected from 200 patients, 100 each with and without breast cancer (40 with benign lesions and 60 without lesions) from one hospital. Ten readers, including five breast specialist radiologists (BSRs) and five general radiologists (GRs), assessed all mammography images using a seven-point scale to rate the likelihood of malignancy in two sessions, with and without the aid of the AI-based software, and the reading time was automatically recorded using a web-based reporting system. Two reading sessions were conducted with a two-month washout period in between. Differences in the area under the receiver operating characteristic curve (AUROC), sensitivity, specificity, and reading time between reading with and without AI were analyzed, accounting for data clustering by readers when indicated.

Results

The AUROC of the AI alone, BSR (average across five readers), and GR (average across five readers) groups was 0.915 (95% confidence interval, 0.876–0.954), 0.813 (0.756–0.870), and 0.684 (0.616–0.752), respectively. With AI assistance, the AUROC significantly increased to 0.884 (0.840–0.928) and 0.833 (0.779–0.887) in the BSR and GR groups, respectively (p = 0.007 and p < 0.001, respectively). Sensitivity was improved by AI assistance in both groups (74.6% vs. 88.6% in BSR, p < 0.001; 52.1% vs. 79.4% in GR, p < 0.001), but the specificity did not differ significantly (66.6% vs. 66.4% in BSR, p = 0.238; 70.8% vs. 70.0% in GR, p = 0.689). The average reading time pooled across readers was significantly decreased by AI assistance for BSRs (82.73 vs. 73.04 seconds, p < 0.001) but increased in GRs (35.44 vs. 42.52 seconds, p < 0.001).

Conclusion

AI-based software improved the performance of radiologists regardless of their experience and affected the reading time.

Artificial intelligence in breast cancer imaging: risk stratification, lesion detection and classification, treatment planning and prognosis-a narrative review

The advent of artificial intelligence (AI) represents a real game changer in today's landscape of breast cancer imaging. Several innovative AI-based tools have been developed and validated in recent years that promise to accelerate the goal of real patient-tailored management. Numerous studies confirm that proper integration of AI into existing clinical workflows could bring significant benefits to women, radiologists, and healthcare systems. The AI-based approach has proved particularly useful for developing new risk prediction models that integrate multi-data streams for planning individualized screening protocols. Furthermore, AI models could help radiologists in the pre-screening and lesion detection phase, increasing diagnostic accuracy, while reducing workload and complications related to overdiagnosis. Radiomics and radiogenomics approaches could extrapolate the so-called imaging signature of the tumor to plan a targeted treatment. The main challenges to the development of AI tools are the huge amounts of high-quality data required to train and validate these models and the need for a multidisciplinary team with solid machine-learning skills. The purpose of this article is to present a summary of the most important AI applications in breast cancer imaging, analyzing possible challenges and new perspectives related to the widespread adoption of these new tools.

Accuracy of ultrasonic artificial intelligence in diagnosing benign and malignant breast diseases: A protocol for systematic review and meta-analysis

BACKGROUND

Artificial intelligence system is a deep learning system based on computer-assisted ultrasonic image diagnosis, which can extract morphological features of breast mass and conduct objective and efficient image analysis, thus automatically intelligent classification of breast mass, avoiding subjective error and improving the accuracy of diagnosis.[1-2] A large number of studies have confirmed that artificial intelligence (AI) has high effectiveness and reliability in the differential diagnosis of benign and malignant breast diseases.[3-4] However, the results of these studies have been contradictory. Therefore, this meta-analysis tested the hypothesis that artificial intelligence system is accurate in distinguishing benign and malignant breast diseases.

METHODS

We will search PubMed, Web of Science, Cochrane Library, and Chinese biomedical databases from their inceptions to the November 20, 2021, without language restrictions. Two authors will independently carry out searching literature records, scanning titles and abstracts, full texts, collecting data, and assessing risk of bias. Review Manager 5.2 and Stata14.0 software will be used for data analysis.

RESULTS

This systematic review will determine the accuracy of AI in the differential diagnosis of benign and malignant breast diseases.

CONCLUSION

Its findings will provide helpful evidence for the accuracy of AI in the differential diagnosis of benign and malignant breast diseases.

SYSTEMATIC REVIEW REGISTRATION

INPLASY2021110087.

Artificial intelligence with deep learning in nuclear medicine and radiology

The use of deep learning in medical imaging has increased rapidly over the past few years, finding applications throughout the entire radiology pipeline, from improved scanner performance to automatic disease detection and diagnosis. These advancements have resulted in a wide variety of deep learning approaches being developed, solving unique challenges for various imaging modalities. This paper provides a review on these developments from a technical point of view, categorizing the different methodologies and summarizing their implementation. We provide an introduction to the design of neural networks and their training procedure, after which we take an extended look at their uses in medical imaging. We cover the different sections of the radiology pipeline, highlighting some influential works and discussing the merits and limitations of deep learning approaches compared to other traditional methods. As such, this review is intended to provide a broad yet concise overview for the interested reader, facilitating adoption and interdisciplinary research of deep learning in the field of medical imaging.

Curation of the CANDID-PTX Dataset with Free-Text Reports

Supplemental material is available for this article. Keywords: Conventional Radiography, Thorax, Trauma, Ribs, Catheters, Segmentation, Diagnosis, Classification, Supervised Learning, Machine Learning © RSNA, 2021.

Artificial Intelligence Detection of Missed Cancers at Digital Mammography That Were Detected at Digital Breast Tomosynthesis

Purpose

To investigate how an artificial intelligence (AI) system performs at digital mammography (DM) from a screening population with ground truth defined by digital breast tomosynthesis (DBT), and whether AI could detect breast cancers at DM that had originally only been detected at DBT.

Materials and Methods

In this secondary analysis of data from a prospective study, DM examinations from 14 768 women (mean age, 57 years), examined with both DM and DBT with independent double reading in the Malmӧ Breast Tomosynthesis Screening Trial (MBTST) (ClinicalTrials.gov: NCT01091545; data collection, 2010-2015), were analyzed with an AI system. Of 136 screening-detected cancers, 95 cancers were detected at DM and 41 cancers were detected only at DBT. The system identifies suspicious areas in the image, scored 1-100, and provides a risk score of 1 to 10 for the whole examination. A cancer was defined as AI detected if the cancer lesion was correctly localized and scored at least 62 (threshold determined by the AI system developers), therefore resulting in the highest examination risk score of 10. Data were analyzed with descriptive statistics, and detection performance was analyzed with receiver operating characteristics.

Results

The highest examination risk score was assigned to 10% (1493 of 14 786) of the examinations. With 90.8% specificity, the AI system detected 75% (71 of 95) of the DM-detected cancers and 44% (18 of 41) of cancers at DM that had originally been detected only at DBT. The majority were invasive cancers (17 of 18).

Conclusion

Almost half of the additional DBT-only screening-detected cancers in the MBTST were detected at DM with AI. AI did not reach double reading performance; however, if combined with double reading, AI has the potential to achieve a substantial portion of the benefit of DBT screening.Keywords: Computer-aided Diagnosis, Mammography, Breast, Diagnosis, Classification, Application DomainClinical trial registration no. NCT01091545© RSNA, 2021.

Classification of MR-Detected Additional Lesions in Patients With Breast Cancer Using a Combination of Radiomics Analysis and Machine Learning

Objective

This study was conducted in order to investigate the feasibility of using radiomics analysis (RA) with machine learning algorithms based on breast magnetic resonance (MR) images for discriminating malignant from benign MR-detected additional lesions in patients with primary breast cancer.

Materials and Methods

One hundred seventy-four MR-detected additional lesions (benign, n = 86; malignancy, n = 88) from 158 patients with ipsilateral primary breast cancer from a tertiary medical center were included in this retrospective study. The entire data were randomly split to training (80%) and independent test sets (20%). In addition, 25 patients (benign, n = 21; malignancy, n = 15) from another tertiary medical center were included for the external test. Radiomics features that were extracted from three regions-of-interest (ROIs; intratumor, peritumor, combined) using fat-saturated T1-weighted images obtained by subtracting pre- from postcontrast images (SUB) and T2-weighted image (T2) were utilized to train the support vector machine for the binary classification. A decision tree method was utilized to build a classifier model using clinical imaging interpretation (CII) features assessed by radiologists. Area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, and specificity were used to compare the diagnostic performance.

Results

The RA models trained using radiomics features from the intratumor-ROI showed comparable performance to the CII model (accuracy, AUROC: 73.3%, 69.6% for the SUB RA model; 70.0%, 75.1% for the T2 RA model; 73.3%, 72.0% for the CII model). The diagnostic performance increased when the radiomics and CII features were combined to build a fusion model. The fusion model that combines the CII features and radiomics features from multiparametric MRI data demonstrated the highest performance with an accuracy of 86.7% and an AUROC of 91.1%. The external test showed a similar pattern where the fusion models demonstrated higher levels of performance compared with the RA- or CII-only models. The accuracy and AUROC of the SUB+T2 RA+CII model in the external test were 80.6% and 91.4%, respectively.

Conclusion

Our study demonstrated the feasibility of using RA with machine learning approach based on multiparametric MRI for quantitatively characterizing MR-detected additional lesions. The fusion model demonstrated an improved diagnostic performance over the models trained with either RA or CII alone.

Regulatory Issues and Challenges to Artificial Intelligence Adoption

Artificial intelligence technology promises to redefine the practice of radiology. However, it exists in a nascent phase and remains largely untested in the clinical space. This nature is both a cause and consequence of the uncertain legal-regulatory environment it enters. This discussion aims to shed light on these challenges, tracing the various pathways toward approval by the US Food and Drug Administration, the future of government oversight, privacy issues, ethical dilemmas, and practical considerations related to implementation in radiologist practice.

Artificial intelligence for the real world of breast screening

Breast cancer screening with mammography reduces mortality in the women who attend by detecting high risk cancer early. It is far from perfect with variations in both sensitivity for the detection of cancer and very wide variations in specificity, leading to unnecessary recalls and biopsies. Over the last 12 months several papers have reported on AI algorithms that perform as well as human readers on large well curated population data sets. The nature of the test sets, the way the gold standard has been calculated, the definition of a positive call, and the statistics used all influence the results. Historically retrospective studies have not predicted the real-life performance of radiologist plus machine. So, it is important to perform prospective studies before introducing Artificial intelligence into real world breast screening.

A Continuously Benchmarked and Crowdsourced Challenge for Rapid Development and Evaluation of Models to Predict COVID-19 Diagnosis and Hospitalization

Importance

Machine learning could be used to predict the likelihood of diagnosis and severity of illness. Lack of COVID-19 patient data has hindered the data science community in developing models to aid in the response to the pandemic.

Objectives

To describe the rapid development and evaluation of clinical algorithms to predict COVID-19 diagnosis and hospitalization using patient data by citizen scientists, provide an unbiased assessment of model performance, and benchmark model performance on subgroups.

Design, Setting, and Participants

This diagnostic and prognostic study operated a continuous, crowdsourced challenge using a model-to-data approach to securely enable the use of regularly updated COVID-19 patient data from the University of Washington by participants from May 6 to December 23, 2020. A postchallenge analysis was conducted from December 24, 2020, to April 7, 2021, to assess the generalizability of models on the cumulative data set as well as subgroups stratified by age, sex, race, and time of COVID-19 test. By December 23, 2020, this challenge engaged 482 participants from 90 teams and 7 countries.

Main Outcomes and Measures

Machine learning algorithms used patient data and output a score that represented the probability of patients receiving a positive COVID-19 test result or being hospitalized within 21 days after receiving a positive COVID-19 test result. Algorithms were evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC) scores. Ensemble models aggregating models from the top challenge teams were developed and evaluated.

Results

In the analysis using the cumulative data set, the best performance for COVID-19 diagnosis prediction was an AUROC of 0.776 (95% CI, 0.775-0.777) and an AUPRC of 0.297, and for hospitalization prediction, an AUROC of 0.796 (95% CI, 0.794-0.798) and an AUPRC of 0.188. Analysis on top models submitting to the challenge showed consistently better model performance on the female group than the male group. Among all age groups, the best performance was obtained for the 25- to 49-year age group, and the worst performance was obtained for the group aged 17 years or younger.

Conclusions and Relevance

In this diagnostic and prognostic study, models submitted by citizen scientists achieved high performance for the prediction of COVID-19 testing and hospitalization outcomes. Evaluation of challenge models on demographic subgroups and prospective data revealed performance discrepancies, providing insights into the potential bias and limitations in the models.

Knowledge and attitudes towards artificial intelligence in imaging: a look at the quantitative survey literature

RATIONALE AND OBJECTIVES

There exists many single sample perspectives on artificial intelligence (AI). The aim of this review was to collate the current data on attitudes/knowledge towards AI in three unique populations: medical students, clinicians and patients.

MATERIALS AND METHODS

A literature search was performed on PubMed, Scopus and Web of Science pertaining to survey data on AI in radiology. Quality assessment was performed by an adapted version of the assessment tool from the National Heart, Lung and Blood Institute for Observational Studies.

RESULTS

Fourteen studies were found on attitudes/knowledge towards AI in radiology. Four studies examined medical students, seven on clinicians and three on patient populations. Deficiencies in the literature mainly related to sampling bias. Students had anxiety relating to future job prospects. Clinicians were optimistic and viewed AI as an aid to the diagnosis and wanted to further their knowledge. Patients were concerned about the lack of human interaction and accountability during error.

CONCLUSION

Attitudes and knowledge regarding AI in radiology remains a topic that needs to be researched further and education given pertaining to its use in a clinical setting.

AI-enhanced breast imaging: Where are we and where are we heading?

Significant advances in imaging analysis and the development of high-throughput methods that can extract and correlate multiple imaging parameters with different clinical outcomes have led to a new direction in medical research. Radiomics and artificial intelligence (AI) studies are rapidly evolving and have many potential applications in breast imaging, such as breast cancer risk prediction, lesion detection and classification, radiogenomics, and prediction of treatment response and clinical outcomes. AI has been applied to different breast imaging modalities, including mammography, ultrasound, and magnetic resonance imaging, in different clinical scenarios. The application of AI tools in breast imaging has an unprecedented opportunity to better derive clinical value from imaging data and reshape the way we care for our patients. The aim of this study is to review the current knowledge and future applications of AI-enhanced breast imaging in clinical practice.

Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy

OBJECTIVE

To examine the accuracy of artificial intelligence (AI) for the detection of breast cancer in mammography screening practice.

DESIGN

Systematic review of test accuracy studies.

DATA SOURCES

Medline, Embase, Web of Science, and Cochrane Database of Systematic Reviews from 1 January 2010 to 17 May 2021.

ELIGIBILITY CRITERIA

Studies reporting test accuracy of AI algorithms, alone or in combination with radiologists, to detect cancer in women's digital mammograms in screening practice, or in test sets. Reference standard was biopsy with histology or follow-up (for screen negative women). Outcomes included test accuracy and cancer type detected.

STUDY SELECTION AND SYNTHESIS

Two reviewers independently assessed articles for inclusion and assessed the methodological quality of included studies using the QUality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. A single reviewer extracted data, which were checked by a second reviewer. Narrative data synthesis was performed.

RESULTS

Twelve studies totalling 131 822 screened women were included. No prospective studies measuring test accuracy of AI in screening practice were found. Studies were of poor methodological quality. Three retrospective studies compared AI systems with the clinical decisions of the original radiologist, including 79 910 women, of whom 1878 had screen detected cancer or interval cancer within 12 months of screening. Thirty four (94%) of 36 AI systems evaluated in these studies were less accurate than a single radiologist, and all were less accurate than consensus of two or more radiologists. Five smaller studies (1086 women, 520 cancers) at high risk of bias and low generalisability to the clinical context reported that all five evaluated AI systems (as standalone to replace radiologist or as a reader aid) were more accurate than a single radiologist reading a test set in the laboratory. In three studies, AI used for triage screened out 53%, 45%, and 50% of women at low risk but also 10%, 4%, and 0% of cancers detected by radiologists.

CONCLUSIONS

Current evidence for AI does not yet allow judgement of its accuracy in breast cancer screening programmes, and it is unclear where on the clinical pathway AI might be of most benefit. AI systems are not sufficiently specific to replace radiologist double reading in screening programmes. Promising results in smaller studies are not replicated in larger studies. Prospective studies are required to measure the effect of AI in clinical practice. Such studies will require clear stopping rules to ensure that AI does not reduce programme specificity.

STUDY REGISTRATION

Protocol registered as PROSPERO CRD42020213590.

The Fermi-Dirac distribution provides a calibrated probabilistic output for binary classifiers

While it would be desirable that the output of binary classification algorithms be the probability that the classification is correct, most algorithms do not provide a method to calculate such a probability. We propose a probabilistic output for binary classifiers based on an unexpected mapping of the probability of correct classification to the probability of occupation of a fermion in a quantum system, known as the Fermi–Dirac distribution. This mapping allows us to compute the optimal threshold to separate predicted classes and to calculate statistical parameters necessary to estimate confidence intervals of performance metrics. Using this mapping we propose an ensemble learning algorithm. In short, the Fermi–Dirac distribution provides a calibrated probabilistic output for binary classification.

Binary classification is one of the central problems in machine-learning research and, as such, investigations of its general statistical properties are of interest. We studied the ranking statistics of items in binary classification problems and observed that there is a formal and surprising relationship between the probability of a sample belonging to one of the two classes and the Fermi–Dirac distribution determining the probability that a fermion occupies a given single-particle quantum state in a physical system of noninteracting fermions. Using this equivalence, it is possible to compute a calibrated probabilistic output for binary classifiers. We show that the area under the receiver operating characteristics curve (AUC) in a classification problem is related to the temperature of an equivalent physical system. In a similar manner, the optimal decision threshold between the two classes is associated with the chemical potential of an equivalent physical system. Using our framework, we also derive a closed-form expression to calculate the variance for the AUC of a classifier. Finally, we introduce FiDEL (Fermi–Dirac-based ensemble learning), an ensemble learning algorithm that uses the calibrated nature of the classifier’s output probability to combine possibly very different classifiers.

Using deep learning to assist readers during the arbitration process: a lesion-based retrospective evaluation of breast cancer screening performance

Objectives

To evaluate if artificial intelligence (AI) can discriminate recalled benign from recalled malignant mammographic screening abnormalities to improve screening performance.

Methods

A total of 2257 full-field digital mammography screening examinations, obtained 2011–2013, of women aged 50–69 years which were recalled for further assessment of 295 malignant out of 305 truly malignant lesions and 2289 benign lesions after independent double-reading with arbitration, were included in this retrospective study. A deep learning AI system was used to obtain a score (0–95) for each recalled lesion, representing the likelihood of breast cancer. The sensitivity on the lesion level and the proportion of women without false-positive ratings (non-FPR) resulting under AI were estimated as a function of the classification cutoff and compared to that of human readers.

Results

Using a cutoff of 1, AI decreased the proportion of women with false-positives from 89.9 to 62.0%, non-FPR 11.1% vs. 38.0% (difference 26.9%, 95% confidence interval 25.1–28.8%; p < .001), preventing 30.1% of reader-induced false-positive recalls, while reducing sensitivity from 96.7 to 91.1% (5.6%, 3.1–8.0%) as compared to human reading. The positive predictive value of recall (PPV-1) increased from 12.8 to 16.5% (3.7%, 3.5–4.0%). In women with mass-related lesions (n = 900), the non-FPR was 14.2% for humans vs. 36.7% for AI (22.4%, 19.8–25.3%) at a sensitivity of 98.5% vs. 97.1% (1.5%, 0–3.5%).

Conclusion

The application of AI during consensus conference might especially help readers to reduce false-positive recalls of masses at the expense of a small sensitivity reduction. Prospective studies are needed to further evaluate the screening benefit of AI in practice.

Key Points

• Integrating the use of artificial intelligence in the arbitration process reduces benign recalls and increases the positive predictive value of recall at the expense of some sensitivity loss.

• Application of the artificial intelligence system to aid the decision to recall a woman seems particularly beneficial for masses, where the system reaches comparable sensitivity to that of the readers, but with considerably reduced false-positives.

• About one-fourth of all recalled malignant lesions are not automatically marked by the system such that their evaluation (AI score) must be retrieved manually by the reader. A thorough reading of screening mammograms by readers to identify suspicious lesions therefore remains mandatory.

A Data Set and Deep Learning Algorithm for the Detection of Masses and Architectural Distortions in Digital Breast Tomosynthesis Images

IMPORTANCE

Breast cancer screening is among the most common radiological tasks, with more than 39 million examinations performed each year. While it has been among the most studied medical imaging applications of artificial intelligence, the development and evaluation of algorithms are hindered by the lack of well-annotated, large-scale publicly available data sets.

OBJECTIVES

To curate, annotate, and make publicly available a large-scale data set of digital breast tomosynthesis (DBT) images to facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening; to develop a baseline deep learning model for breast cancer detection; and to test this model using the data set to serve as a baseline for future research.

DESIGN, SETTING, AND PARTICIPANTS

In this diagnostic study, 16 802 DBT examinations with at least 1 reconstruction view available, performed between August 26, 2014, and January 29, 2018, were obtained from Duke Health System and analyzed. From the initial cohort, examinations were divided into 4 groups and split into training and test sets for the development and evaluation of a deep learning model. Images with foreign objects or spot compression views were excluded. Data analysis was conducted from January 2018 to October 2020.

EXPOSURES

Screening DBT.

MAIN OUTCOMES AND MEASURES

The detection algorithm was evaluated with breast-based free-response receiver operating characteristic curve and sensitivity at 2 false positives per volume.

RESULTS

The curated data set contained 22 032 reconstructed DBT volumes that belonged to 5610 studies from 5060 patients with a mean (SD) age of 55 (11) years and 5059 (100.0%) women. This included 4 groups of studies: (1) 5129 (91.4%) normal studies; (2) 280 (5.0%) actionable studies, for which where additional imaging was needed but no biopsy was performed; (3) 112 (2.0%) benign biopsied studies; and (4) 89 studies (1.6%) with cancer. Our data set included masses and architectural distortions that were annotated by 2 experienced radiologists. Our deep learning model reached breast-based sensitivity of 65% (39 of 60; 95% CI, 56%-74%) at 2 false positives per DBT volume on a test set of 460 examinations from 418 patients.

CONCLUSIONS AND RELEVANCE

The large, diverse, and curated data set presented in this study could facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening by providing data for training as well as a common set of cases for model validation. The performance of the model developed in this study showed that the task remains challenging; its performance could serve as a baseline for future model development.

Machine learning and big data provide crucial insight for future biomaterials discovery and research

Machine learning have been widely adopted in a variety of fields including engineering, science, and medicine revolutionizing how data is collected, used, and stored. Their implementation has led to a drastic increase in the number of computational models for the prediction of various numerical, categorical, or association events given input variables. We aim to examine recent advances in the use of machine learning when applied to the biomaterial field. Specifically, quantitative structure properties relationships offer the unique ability to correlate microscale molecular descriptors to larger macroscale material properties. These new models can be broken down further into four categories: regression, classification, association, and clustering. We examine recent approaches and new uses of machine learning in the three major categories of biomaterials: metals, polymers, and ceramics for rapid property prediction and trend identification. While current research is promising, limitations in the form of lack of standardized reporting and available databases complicates the implementation of described models. Herein, we hope to provide a snapshot of the current state of the field and a beginner's guide to navigating the intersection of biomaterials research and machine learning. STATEMENT OF SIGNIFICANCE: Machine learning and its methods have found a variety of uses beyond the field of computer science but have largely been neglected by those in realm of biomaterials. Through the use of more computational methods, biomaterials development can be expediated while reducing the need for standard trial and error methods. Within, we introduce four basic models that readers can potentially apply to their current research as well as current applications within the field. Furthermore, we hope that this article may act as a "call to action" for readers to realize and address the current lack of implementation within the biomaterials field.

Toward robust mammography-based models for breast cancer risk

Improved breast cancer risk models enable targeted screening strategies that achieve earlier detection and less screening harm than existing guidelines. To bring deep learning risk models to clinical practice, we need to further refine their accuracy, validate them across diverse populations, and demonstrate their potential to improve clinical workflows. We developed Mirai, a mammography-based deep learning model designed to predict risk at multiple timepoints, leverage potentially missing risk factor information, and produce predictions that are consistent across mammography machines. Mirai was trained on a large dataset from Massachusetts General Hospital (MGH) in the United States and tested on held-out test sets from MGH, Karolinska University Hospital in Sweden, and Chang Gung Memorial Hospital (CGMH) in Taiwan, obtaining C-indices of 0.76 (95% confidence interval, 0.74 to 0.80), 0.81 (0.79 to 0.82), and 0.79 (0.79 to 0.83), respectively. Mirai obtained significantly higher 5-year ROC AUCs than the Tyrer-Cuzick model ( P < 0.001) and prior deep learning models Hybrid DL ( P < 0.001) and Image-Only DL ( P < 0.001), trained on the same dataset. Mirai more accurately identified high-risk patients than prior methods across all datasets. On the MGH test set, 41.5% (34.4 to 48.5) of patients who would develop cancer within 5 years were identified as high risk, compared with 36.1% (29.1 to 42.9) by Hybrid DL ( P = 0.02) and 22.9% (15.9 to 29.6) by the Tyrer-Cuzick model ( P < 0.001).

Comparison of the Predicting Performance for Fate of Medial Meniscus Posterior Root Tear Based on Treatment Strategies: A Comparison between Logistic Regression, Gradient Boosting, and CNN Algorithms

This study aimed to validate the accuracy and prediction performance of machine learning (ML), deep learning (DL), and logistic regression methods in the treatment of medial meniscus posterior root tears (MMPRT). From July 2003 to May 2018, 640 patients diagnosed with MMPRT were included. First, the affecting factors for the surgery were evaluated using statistical analysis. Second, AI technology was introduced using X-ray and MRI. Finally, the accuracy and prediction performance were compared between ML&DL and logistic regression methods. Affecting factors of the logistic regression method corresponded well with the feature importance of the six top-ranked factors in the ML&DL method. There was no significant difference when comparing the accuracy, F1-score, and error rate between ML&DL and logistic regression methods (accuracy = 0.89 and 0.91, F1 score = 0.89 and 0.90, error rate = 0.11 and 0.09; p = 0.114, 0.422, and 0.119, respectively). The area under the curve (AUC) values showed excellent test quality for both ML&DL and logistic regression methods (AUC = 0.97 and 0.94, respectively) in the evaluation of prediction performance (p = 0.289). The affecting factors of the logistic regression method and the influence of the ML&DL method were not significantly different. The accuracy and performance of the ML&DL method in predicting the fate of MMPRT were comparable to those of the logistic regression method. Therefore, this ML&DL algorithm could potentially predict the outcome of the MMRPT in various fields and situations. Furthermore, our method could be efficiently implemented in current clinical practice.

Evaluation of deep learning-based artificial intelligence techniques for breast cancer detection on mammograms: Results from a retrospective study using a BreastScreen Victoria dataset

Introduction

This study aims to evaluate deep learning (DL)‐based artificial intelligence (AI) techniques for detecting the presence of breast cancer on a digital mammogram image.

Methods

We evaluated several DL‐based AI techniques that employ different approaches and backbone DL models and tested the effect on performance of using different data‐processing strategies on a set of digital mammographic images with annotations of pathologically proven breast cancer.

Results

Our evaluation uses the area under curve (AUC) and accuracy (ACC) for performance measurement. The best evaluation result, based on 349 test cases (930 test images), was an AUC of 0.8979 [95% confidence interval (CI) 0.873, 0.923] and ACC of 0.8178 [95% CI 0.785, 0.850]. This was achieved by an AI technique that utilises a certain family of DL models, namely ResNet, as its backbone, combines the global features extracted from the whole mammogram and the local features extracted from the automatically detected cancer and non‐cancer local regions in the whole image, and leverages background cropping and text removal, contrast adjustment and more training data.

Conclusion

DL‐based AI techniques have shown promising results in retrospective studies for many medical image analysis applications. Our study demonstrates a significant opportunity to boost the performance of such techniques applied to breast cancer detection by exploring different types of approaches, backbone DL models and data‐processing strategies. The promising results we have obtained suggest further development of AI reading services could transform breast cancer screening in the future.

Adoption of artificial intelligence in breast imaging: evaluation, ethical constraints and limitations

Retrospective studies have shown artificial intelligence (AI) algorithms can match as well as enhance radiologist's performance in breast screening. These tools can facilitate tasks not feasible by humans such as the automatic triage of patients and prediction of treatment outcomes. Breast imaging faces growing pressure with the exponential growth in imaging requests and a predicted reduced workforce to provide reports. Solutions to alleviate these pressures are being sought with an increasing interest in the adoption of AI to improve workflow efficiency as well as patient outcomes. Vast quantities of data are needed to test and monitor AI algorithms before and after their incorporation into healthcare systems. Availability of data is currently limited, although strategies are being devised to harness the data that already exists within healthcare institutions. Challenges that underpin the realisation of AI into everyday breast imaging cannot be underestimated and the provision of guidance from national agencies to tackle these challenges, taking into account views from a societal, industrial and healthcare prospective is essential. This review provides background on the evaluation and use of AI in breast imaging in addition to exploring key ethical, technical, legal and regulatory challenges that have been identified so far.

Editorial: Artificial Intelligence (AI) in Clinical Medicine and the 2020 CONSORT-AI Study Guidelines

Artificial intelligence (AI) in clinical medicine includes physical robotics and devices and virtual AI and machine learning. Concerns have been raised regarding ethical issues for the use of AI in surgery, including guidance for surgical decisions, patient confidentiality, and the need for support from controlled clinical trials to use these methods so that clinical guidelines can be developed. The most common applications for virtual AI include disease diagnosis, health monitoring and digital patient consultations, clinical training, patient data management, drug development, and personalized medicine. In September 2020, the CONSORT-A1 extension was developed with 14 additional items that should be reported for AI studies that include clear descriptions of the AI intervention, skills required, study setting, inputs and outputs of the AI intervention, analysis of errors, and the human and AI interactions. This Editorial aims to present current applications and challenges of AI in clinical medicine and the importance of the new 2020 CONSORT-AI study guidelines.

Rethinking the Approach to Artificial Intelligence for Medical Image Analysis: The Case for Precision Diagnosis

To date, widely generalizable artificial intelligence (AI) programs for medical image analysis have not been demonstrated, including for mammography. Rather than pursuing a strategy of collecting ever-larger databases in the attempt to build generalizable programs, we suggest three possible avenues for exploring a precision medicine or precision imaging approach. First, it is now technologically feasible to collect hundreds of thousands of multi-institutional cases along with other patient data, allowing stratification of patients into subpopulations that have similar characteristics in the manner discussed by the National Research Council in its white paper on precision medicine. A family of AI programs could be developed across different examination types that are matched to specific patient subpopulations. Such stratification can help address bias, including racial or ethnic bias, by allowing unbiased data aggregation for creation of subpopulations. Second, for common examinations, larger institutions may be able to collect enough of their own data to train AI programs that reflect disease prevalence and variety in their respective unique patient subpopulations. Third, high- and low-probability subpopulations can be identified by application of AI programs, thereby allowing their triage off the radiology work list. This would reduce radiologists' workloads, providing more time for interpretation of the remaining examinations. For high-volume procedures, investigators should come together to define reference standards, collect data, and compare the merits of pursuing generalizability versus a precision medicine subpopulation-based strategy.

Brain MRI Deep Learning and Bayesian Inference System Augments Radiology Resident Performance

Automated quantitative and probabilistic medical image analysis has the potential to improve the accuracy and efficiency of the radiology workflow. We sought to determine whether AI systems for brain MRI diagnosis could be used as a clinical decision support tool to augment radiologist performance. We utilized previously developed AI systems that combine convolutional neural networks and expert-derived Bayesian networks to distinguish among 50 diagnostic entities on multimodal brain MRIs. We tested whether these systems could augment radiologist performance through an interactive clinical decision support tool known as Adaptive Radiology Interpretation and Education System (ARIES) in 194 test cases. Four radiology residents and three academic neuroradiologists viewed half of the cases unassisted and half with the results of the AI system displayed on ARIES. Diagnostic accuracy of radiologists for top diagnosis (TDx) and top three differential diagnosis (T3DDx) was compared with and without ARIES. Radiology resident performance was significantly better with ARIES for both TDx (55% vs 30%; P < .001) and T3DDx (79% vs 52%; P = 0.002), with the largest improvement for rare diseases (39% increase for T3DDx; P < 0.001). There was no significant difference between attending performance with and without ARIES for TDx (72% vs 69%; P = 0.48) or T3DDx (86% vs 89%; P = 0.39). These findings suggest that a hybrid deep learning and Bayesian inference clinical decision support system has the potential to augment diagnostic accuracy of non-specialists to approach the level of subspecialists for a large array of diseases on brain MRI.

Crowdsourcing assessment of maternal blood multi-omics for predicting gestational age and preterm birth

Identification of pregnancies at risk of preterm birth (PTB), the leading cause of newborn deaths, remains challenging given the syndromic nature of the disease. We report a longitudinal multi-omics study coupled with a DREAM challenge to develop predictive models of PTB. The findings indicate that whole-blood gene expression predicts ultrasound-based gestational ages in normal and complicated pregnancies (r = 0.83) and, using data collected before 37 weeks of gestation, also predicts the delivery date in both normal pregnancies (r = 0.86) and those with spontaneous preterm birth (r = 0.75). Based on samples collected before 33 weeks in asymptomatic women, our analysis suggests that expression changes preceding preterm prelabor rupture of the membranes are consistent across time points and cohorts and involve leukocyte-mediated immunity. Models built from plasma proteomic data predict spontaneous preterm delivery with intact membranes with higher accuracy and earlier in pregnancy than transcriptomic models (AUROC = 0.76 versus AUROC = 0.6 at 27-33 weeks of gestation).

Artificial intelligence in breast cancer screening: primary care provider preferences

BACKGROUND

Artificial intelligence (AI) is increasingly being proposed for use in medicine, including breast cancer screening (BCS). Little is known, however, about referring primary care providers' (PCPs') preferences for this technology.

METHODS

We identified the most important attributes of AI BCS for ordering PCPs using qualitative interviews: sensitivity, specificity, radiologist involvement, understandability of AI decision-making, supporting evidence, and diversity of training data. We invited US-based PCPs to participate in an internet-based experiment designed to force participants to trade off among the attributes of hypothetical AI BCS products. Responses were analyzed with random parameters logit and latent class models to assess how different attributes affect the choice to recommend AI-enhanced screening.

RESULTS

Ninety-one PCPs participated. Sensitivity was most important, and most PCPs viewed radiologist participation in mammography interpretation as important. Other important attributes were specificity, understandability of AI decision-making, and diversity of data. We identified 3 classes of respondents: "Sensitivity First" (41%) found sensitivity to be more than twice as important as other attributes; "Against AI Autonomy" (24%) wanted radiologists to confirm every image; "Uncertain Trade-Offs" (35%) viewed most attributes as having similar importance. A majority (76%) accepted the use of AI in a "triage" role that would allow it to filter out likely negatives without radiologist confirmation.

CONCLUSIONS AND RELEVANCE

Sensitivity was the most important attribute overall, but other key attributes should be addressed to produce clinically acceptable products. We also found that most PCPs accept the use of AI to make determinations about likely negative mammograms without radiologist confirmation.

On the role of artificial intelligence in medical imaging of COVID-19

Although a plethora of research articles on AI methods on COVID-19 medical imaging are published, their clinical value remains unclear. We conducted the largest systematic review of the literature addressing the utility of AI in imaging for COVID-19 patient care. By keyword searches on PubMed and preprint servers throughout 2020, we identified 463 manuscripts and performed a systematic meta-analysis to assess their technical merit and clinical relevance. Our analysis evidences a significant disparity between clinical and AI communities, in the focus on both imaging modalities (AI experts neglected CT and ultrasound, favoring X-ray) and performed tasks (71.9% of AI papers centered on diagnosis). The vast majority of manuscripts were found to be deficient regarding potential use in clinical practice, but 2.7% (n = 12) publications were assigned a high maturity level and are summarized in greater detail. We provide an itemized discussion of the challenges in developing clinically relevant AI solutions with recommendations and remedies.

Advances in systems biology modeling: 10 years of crowdsourcing DREAM challenges

Computational and mathematical models are key to obtain a system-level understanding of biological processes, but their limitations have to be clearly defined to allow their proper application and interpretation. Crowdsourced benchmarks in the form of challenges provide an unbiased assessment of methods, and for the past decade, the Dialogue for Reverse Engineering Assessment and Methods (DREAM) organized more than 15 systems biology challenges. From transcription factor binding to dynamical network models, from signaling networks to gene regulation, from whole-cell models to cell-lineage reconstruction, and from single-cell positioning in a tissue to drug combinations and cell survival, the breadth is broad. To celebrate the 5-year anniversary of Cell Systems, we review the genesis of these systems biology challenges and discuss how interlocking the forward- and reverse-modeling paradigms allows to push the rim of systems biology. This approach will persist for systems levels approaches in biology and medicine.

Radiomics: A Primer for Breast Radiologists

Radiomics has a long-standing history in breast imaging with computer-aided detection (CAD) for screening mammography developed in the late 20th century. Although conventional CAD had widespread adoption, the clinical benefits for experienced breast radiologists were debatable due to high false-positive marks and subsequent increased recall rates. The dramatic growth in recent years of artificial intelligence-based analysis, including machine learning and deep learning, has provided numerous opportunities for improved modern radiomics work in breast imaging. There has been extensive radiomics work in mammography, digital breast tomosynthesis, MRI, ultrasound, PET-CT, and combined multimodality imaging. Specific radiomics outcomes of interest have been diverse, including CAD, prediction of response to neoadjuvant therapy, lesion classification, and survival, among other outcomes. Additionally, the radiogenomics subfield that correlates radiomics features with genetics has been very proliferative, in parallel with the clinical validation of breast cancer molecular subtypes and gene expression assays. Despite the promise of radiomics, there are important challenges related to image normalization, limited large unbiased data sets, and lack of external validation. Much of the radiomics work to date has been exploratory using single-institution retrospective series for analysis, but several promising lines of investigation have made the leap to clinical practice with commercially available products. As a result, breast radiologists will increasingly be incorporating radiomics-based tools into their daily practice in the near future. Therefore, breast radiologists must have a broad understanding of the scope, applications, and limitations of radiomics work.