Deep learning-enabled medical computer vision

Benchmarking YOLOv5 and YOLOv7 models with DeepSORT for droplet tracking applications

Tracking droplets in microfluidics is a challenging task. The difficulty arises in choosing a tool to analyze general microfluidic videos to infer physical quantities. The state-of-the-art object detector algorithm You Only Look Once (YOLO) and the object tracking algorithm Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) are customizable for droplet identification and tracking. The customization includes training YOLO and DeepSORT networks to identify and track the objects of interest. We trained several YOLOv5 and YOLOv7 models and the DeepSORT network for droplet identification and tracking from microfluidic experimental videos. We compare the performance of the droplet tracking applications with YOLOv5 and YOLOv7 in terms of training time and time to analyze a given video across various hardware configurations. Despite the latest YOLOv7 being 10% faster, the real-time tracking is only achieved by lighter YOLO models on RTX 3070 Ti GPU machine due to additional significant droplet tracking costs arising from the DeepSORT algorithm. This work is a benchmark study for the YOLOv5 and YOLOv7 networks with DeepSORT in terms of the training time and inference time for a custom dataset of microfluidic droplets.

A survey on molecular-scale learning systems with relevance to DNA computing

DNA computing has emerged as a promising alternative to achieve programmable behaviors in chemistry by repurposing the nucleic acid molecules into chemical hardware upon which synthetic chemical programs can be executed. These chemical programs are capable of simulating diverse behaviors, including boolean logic computation, oscillations, and nanorobotics. Chemical environments such as the cell are marked by uncertainty and are prone to random fluctuations. For this reason, potential DNA-based molecular devices that aim to be deployed into such environments should be capable of adapting to the stochasticity inherent in them. In keeping with this goal, a new subfield has emerged within DNA computing, focusing on developing approaches that embed learning and inference into chemical reaction systems. If realized in biochemical contexts, such molecular machines can engender novel applications in fields such as biotechnology, synthetic biology, and medicine. Therefore, it would be beneficial to review how different ideas were conceived, how the progress has been so far, and what the emerging ideas are in this nascent field of 'molecular-scale learning'.

Skin-interfaced electronics: A promising and intelligent paradigm for personalized healthcare

Skin-interfaced electronics (skintronics) have received considerable attention due to their thinness, skin-like mechanical softness, excellent conformability, and multifunctional integration. Current advancements in skintronics have enabled health monitoring and digital medicine. Particularly, skintronics offer a personalized platform for early-stage disease diagnosis and treatment. In this comprehensive review, we discuss (1) the state-of-the-art skintronic devices, (2) material selections and platform considerations of future skintronics toward intelligent healthcare, (3) device fabrication and system integrations of skintronics, (4) an overview of the skintronic platform for personalized healthcare applications, including biosensing as well as wound healing, sleep monitoring, the assessment of SARS-CoV-2, and the augmented reality-/virtual reality-enhanced human-machine interfaces, and (5) current challenges and future opportunities of skintronics and their potentials in clinical translation and commercialization. The field of skintronics will not only minimize physical and physiological mismatches with the skin but also shift the paradigm in intelligent and personalized healthcare and offer unprecedented promise to revolutionize conventional medical practices.

IoT-Enabled Gait Assessment: The Next Step for Habitual Monitoring

Walking/gait quality is a useful clinical tool to assess general health and is now broadly described as the sixth vital sign. This has been mediated by advances in sensing technology, including instrumented walkways and three-dimensional motion capture. However, it is wearable technology innovation that has spawned the highest growth in instrumented gait assessment due to the capabilities for monitoring within and beyond the laboratory. Specifically, instrumented gait assessment with wearable inertial measurement units (IMUs) has provided more readily deployable devices for use in any environment. Contemporary IMU-based gait assessment research has shown evidence of the robust quantifying of important clinical gait outcomes in, e.g., neurological disorders to gather more insightful habitual data in the home and community, given the relatively low cost and portability of IMUs. The aim of this narrative review is to describe the ongoing research regarding the need to move gait assessment out of bespoke settings into habitual environments and to consider the shortcomings and inefficiencies that are common within the field. Accordingly, we broadly explore how the Internet of Things (IoT) could better enable routine gait assessment beyond bespoke settings. As IMU-based wearables and algorithms mature in their corroboration with alternate technologies, such as computer vision, edge computing, and pose estimation, the role of IoT communication will enable new opportunities for remote gait assessment.

Platelet-Based Liquid Biopsies through the Lens of Machine Learning

Liquid biopsies offer minimally invasive diagnosis and monitoring of cancer disease. This biosource is often analyzed using sequencing, which generates highly complex data that can be used using machine learning tools. Nevertheless, validating the clinical applications of such methods is challenging. It requires: (a) using data from many patients; (b) verifying potential bias concerning sample collection; and (c) adding interpretability to the model. In this work, we have used RNA sequencing data of tumor-educated platelets (TEPs) and performed a binary classification (cancer vs. no-cancer). First, we compiled a large-scale dataset with more than a thousand donors. Further, we used different convolutional neural networks (CNNs) and boosting methods to evaluate the classifier performance. We have obtained an impressive result of 0.96 area under the curve. We then identified different clusters of splice variants using expert knowledge from the Kyoto Encyclopedia of Genes and Genomes (KEGG). Employing boosting algorithms, we identified the features with the highest predictive power. Finally, we tested the robustness of the models using test data from novel hospitals. Notably, we did not observe any decrease in model performance. Our work proves the great potential of using TEP data for cancer patient classification and opens the avenue for profound cancer diagnostics.

Geriatric Care Management System Powered by the IoT and Computer Vision Techniques

The digitalisation of geriatric care refers to the use of emerging technologies to manage and provide person-centered care to the elderly by collecting patients' data electronically and using them to streamline the care process, which improves the overall quality, accuracy, and efficiency of healthcare. In many countries, healthcare providers still rely on the manual measurement of bioparameters, inconsistent monitoring, and paper-based care plans to manage and deliver care to elderly patients. This can lead to a number of problems, including incomplete and inaccurate record-keeping, errors, and delays in identifying and resolving health problems. The purpose of this study is to develop a geriatric care management system that combines signals from various wearable sensors, noncontact measurement devices, and image recognition techniques to monitor and detect changes in the health status of a person. The system relies on deep learning algorithms and the Internet of Things (IoT) to identify the patient and their six most pertinent poses. In addition, the algorithm has been developed to monitor changes in the patient's position over a longer period of time, which could be important for detecting health problems in a timely manner and taking appropriate measures. Finally, based on expert knowledge and a priori rules integrated in a decision tree-based model, the automated final decision on the status of nursing care plan is generated to support nursing staff.

The learning curve of bilateral laparoscopic varicocelectomy: a prospective study

Varicocele occurs in 15% of general male population and in 35% of infertile men. Since 1992, surgical correction with laparoscopic varicocelectomy is the gold standard for treatment of symptomatic patients or patients with abnormal seminal analysis. The learning curve for this frequently performed procedure has not yet been described. In the present study, we investigated the learning curve of a single urologist in training performing his first 21 laparoscopic varicocelectomies using qualitative and quantitative tools to evaluate his performance during the process. Our results show that 14 bilateral laparoscopic varicocelectomies are enough to achieve the plateau of the learning curve.

Collaborative strategies for deploying artificial intelligence to complement physician diagnoses of acute respiratory distress syndrome

There is a growing gap between studies describing the capabilities of artificial intelligence (AI) diagnostic systems using deep learning versus efforts to investigate how or when to integrate AI systems into a real-world clinical practice to support physicians and improve diagnosis. To address this gap, we investigate four potential strategies for AI model deployment and physician collaboration to determine their potential impact on diagnostic accuracy. As a case study, we examine an AI model trained to identify findings of the acute respiratory distress syndrome (ARDS) on chest X-ray images. While this model outperforms physicians at identifying findings of ARDS, there are several reasons why fully automated ARDS detection may not be optimal nor feasible in practice. Among several collaboration strategies tested, we find that if the AI model first reviews the chest X-ray and defers to a physician if it is uncertain, this strategy achieves a higher diagnostic accuracy (0.869, 95% CI 0.835-0.903) compared to a strategy where a physician reviews a chest X-ray first and defers to an AI model if uncertain (0.824, 95% CI 0.781-0.862), or strategies where the physician reviews the chest X-ray alone (0.808, 95% CI 0.767-0.85) or the AI model reviews the chest X-ray alone (0.847, 95% CI 0.806-0.887). If the AI model reviews a chest X-ray first, this allows the AI system to make decisions for up to 79% of cases, letting physicians focus on the most challenging subsets of chest X-rays.

Qureshi A, Li J, Mahmood T. On the Analyses of Medical Images Using Traditional Machine Learning Techniques and Convolutional Neural Networks

Convolutional neural network (CNN) has shown dissuasive accomplishment on different areas especially Object Detection, Segmentation, Reconstruction (2D and 3D), Information Retrieval, Medical Image Registration, Multi-lingual translation, Local language Processing, Anomaly Detection on video and Speech Recognition. CNN is a special type of Neural Network, which has compelling and effective learning ability to learn features at several steps during augmentation of the data. Recently, different interesting and inspiring ideas of Deep Learning (DL) such as different activation functions, hyperparameter optimization, regularization, momentum and loss functions has improved the performance, operation and execution of CNN Different internal architecture innovation of CNN and different representational style of CNN has significantly improved the performance. This survey focuses on internal taxonomy of deep learning, different models of vonvolutional neural network, especially depth and width of models and in addition CNN components, applications and current challenges of deep learning.

Interpretation and Use of Applied/Operational Machine Learning and Artificial Intelligence in Surgery

Applications for artificial intelligence (AI) and machine learning in surgery include image interpretation, data summarization, automated narrative construction, trajectory and risk prediction, and operative navigation and robotics. The pace of development has been exponential, and some AI applications are working well. However, demonstrations of clinical utility, validity, and equity have lagged algorithm development and limited widespread adoption of AI into clinical practice. Outdated computing infrastructure and regulatory challenges which promote data silos are key barriers. Multidisciplinary teams will be needed to address these challenges and to build AI systems that are relevant, equitable, and dynamic.

Deep learning for reconstructing protein structures from cryo-EM density maps: Recent advances and future directions

Cryo-Electron Microscopy (cryo-EM) has emerged as a key technology to determine the structure of proteins, particularly large protein complexes and assemblies in recent years. A key challenge in cryo-EM data analysis is to automatically reconstruct accurate protein structures from cryo-EM density maps. In this review, we briefly overview various deep learning methods for building protein structures from cryo-EM density maps, analyze their impact, and discuss the challenges of preparing high-quality data sets for training deep learning models. Looking into the future, more advanced deep learning models of effectively integrating cryo-EM data with other sources of complementary data such as protein sequences and AlphaFold-predicted structures need to be developed to further advance the field.

A compact review of progress and prospects of deep learning in drug discovery

BACKGROUND

Drug discovery processes, such as new drug development, drug synergy, and drug repurposing, consume significant yearly resources. Computer-aided drug discovery can effectively improve the efficiency of drug discovery. Traditional computer methods such as virtual screening and molecular docking have achieved many gratifying results in drug development. However, with the rapid growth of computer science, data structures have changed considerably; with more extensive and dimensional data and more significant amounts of data, traditional computer methods can no longer be applied well. Deep learning methods are based on deep neural network structures that can handle high-dimensional data very well, so they are used in current drug development.

RESULTS

This review summarized the applications of deep learning methods in drug discovery, such as drug target discovery, drug de novo design, drug recommendation, drug synergy, and drug response prediction. While applying deep learning methods to drug discovery suffers from a lack of data, transfer learning is an excellent solution to this problem. Furthermore, deep learning methods can extract deeper features and have higher predictive power than other machine learning methods. Deep learning methods have great potential in drug discovery and are expected to facilitate drug discovery development.

Deep learning-based recognition of key anatomical structures during robot-assisted minimally invasive esophagectomy

OBJECTIVE

To develop a deep learning algorithm for anatomy recognition in thoracoscopic video frames from robot-assisted minimally invasive esophagectomy (RAMIE) procedures using deep learning.

BACKGROUND

RAMIE is a complex operation with substantial perioperative morbidity and a considerable learning curve. Automatic anatomy recognition may improve surgical orientation and recognition of anatomical structures and might contribute to reducing morbidity or learning curves. Studies regarding anatomy recognition in complex surgical procedures are currently lacking.

METHODS

Eighty-three videos of consecutive RAMIE procedures between 2018 and 2022 were retrospectively collected at University Medical Center Utrecht. A surgical PhD candidate and an expert surgeon annotated the azygos vein and vena cava, aorta, and right lung on 1050 thoracoscopic frames. 850 frames were used for training of a convolutional neural network (CNN) to segment the anatomical structures. The remaining 200 frames of the dataset were used for testing the CNN. The Dice and 95% Hausdorff distance (95HD) were calculated to assess algorithm accuracy.

RESULTS

The median Dice of the algorithm was 0.79 (IQR = 0.20) for segmentation of the azygos vein and/or vena cava. A median Dice coefficient of 0.74 (IQR = 0.86) and 0.89 (IQR = 0.30) were obtained for segmentation of the aorta and lung, respectively. Inference time was 0.026 s (39 Hz). The prediction of the deep learning algorithm was compared with the expert surgeon annotations, showing an accuracy measured in median Dice of 0.70 (IQR = 0.19), 0.88 (IQR = 0.07), and 0.90 (0.10) for the vena cava and/or azygos vein, aorta, and lung, respectively.

CONCLUSION

This study shows that deep learning-based semantic segmentation has potential for anatomy recognition in RAMIE video frames. The inference time of the algorithm facilitated real-time anatomy recognition. Clinical applicability should be assessed in prospective clinical studies.

Predicting Drug Synergy and Discovering New Drug Combinations Based on a Graph Autoencoder and Convolutional Neural Network

Drug synergy is a crucial component in drug reuse since it solves the problem of sluggish drug development and the absence of corresponding drugs for several diseases. Predicting drug synergistic relationships can screen drug combinations in advance and reduce the waste of laboratory resources. In this research, we proposed a model that utilizes graph autoencoder and convolutional neural networks to predict drug synergy (GAECDS). Our methods include a graph convolutional neural network as an encoder to encode drug features and use a matrix factorization method as a decoder. Multilayer perceptron (MLP) was applied to process cell line features and combine them with drug features. Furthermore, the latent vectors generated during the encoding process are being used to predict drug synergistic scores using a convolutional neural network. By measuring prediction performance using AUC, AUPR, and F1 score, GAECDS superior to other state-of-the-art models. In addition, four pairs of the predicted top 10 drug combinations were found to work well enough for evaluation. The case study shows that the GAECDS approach is useful for identifying potential drug synergy.

Spatial profiling technologies illuminate the tumor microenvironment

The tumor microenvironment (TME) is composed of many different cellular and acellular components that together drive tumor growth, invasion, metastasis, and response to therapies. Increasing realization of the significance of the TME in cancer biology has shifted cancer research from a cancer-centric model to one that considers the TME as a whole. Recent technological advancements in spatial profiling methodologies provide a systematic view and illuminate the physical localization of the components of the TME. In this review, we provide an overview of major spatial profiling technologies. We present the types of information that can be extracted from these data and describe their applications, findings and challenges in cancer research. Finally, we provide a future perspective of how spatial profiling could be integrated into cancer research to improve patient diagnosis, prognosis, stratification to treatment and development of novel therapeutics.

Artificial intelligence in endoscopic imaging for detection of malignant biliary strictures and cholangiocarcinoma: a systematic review

Background

Artificial intelligence (AI), when applied to computer vision using a convolutional neural network (CNN), is a promising tool in "difficult-to-diagnose" conditions such as malignant biliary strictures and cholangiocarcinoma (CCA). The aim of this systematic review is to summarize and review the available data on the diagnostic utility of endoscopic AI-based imaging for malignant biliary strictures and CCA.

Methods

In this systematic review, PubMed, Scopus and Web of Science databases were reviewed for studies published from January 2000 to June 2022. Extracted data included type of endoscopic imaging modality, AI classifiers, and performance measures.

Results

The search yielded 5 studies involving 1465 patients. Of the 5 included studies, 4 (n=934; 3,775,819 images) used CNN in combination with cholangioscopy, while one study (n=531; 13,210 images) used CNN with endoscopic ultrasound (EUS). The average image processing speed of CNN with cholangioscopy was 7-15 msec per frame while that of CNN with EUS was 200-300 msec per frame. The highest performance metrics were observed with CNN-cholangioscopy (accuracy 94.9%, sensitivity 94.7%, and specificity 92.1%). CNN-EUS was associated with the greatest clinical performance application, providing station recognition and bile duct segmentation; thus reducing procedure length and providing real-time feedback to the endoscopist.

Conclusions

Our results suggest that there is increasing evidence to support a role for AI in the diagnosis of malignant biliary strictures and CCA. CNN-based machine leaning of cholangioscopy images appears to be the most promising, while CNN-EUS has the best clinical performance application.

Deep learning-based image analysis of eyelid morphology in thyroid-associated ophthalmopathy

Background

We aimed to propose a deep learning-based approach to automatically measure eyelid morphology in patients with thyroid-associated ophthalmopathy (TAO).

Methods

This prospective study consecutively included 74 eyes of patients with TAO and 74 eyes of healthy volunteers visiting the ophthalmology department in a tertiary hospital. Patients diagnosed as TAO and healthy volunteers who were age- and gender-matched met the eligibility criteria for recruitment. Facial images were taken under the same light conditions. Comprehensive eyelid morphological parameters, such as palpebral fissure (PF) length, margin reflex distance (MRD), eyelid retraction distance, eyelid length, scleral area, and mid-pupil lid distance (MPLD), were automatically calculated using our deep learning-based analysis system. MRD1 and 2 were manually measured. Bland-Altman plots and intraclass correlation coefficients (ICCs) were performed to assess the agreement between automatic and manual measurements of MRDs. The asymmetry of the eyelid contour was analyzed using the temporal: nasal ratio of the MPLD. All eyelid features were compared between TAO eyes and control eyes using the independent samples t-test.

Results

A strong agreement between automatic and manual measurement was indicated. Biases of MRDs in TAO eyes and control eyes ranged from -0.01 mm [95% limits of agreement (LoA): -0.64 to 0.63 mm] to 0.09 mm (LoA: -0.46 to 0.63 mm). ICCs ranged from 0.932 to 0.980 (P<0.001). Eyelid features were significantly different in TAO eyes and control eyes, including MRD1 (4.82±1.59 vs. 2.99±0.81 mm; P<0.001), MRD2 (5.89±1.16 vs. 5.47±0.73 mm; P=0.009), upper eyelid length (UEL) (27.73±4.49 vs. 25.42±4.35 mm; P=0.002), lower eyelid length (LEL) (31.51±4.59 vs. 26.34±4.72 mm; P<0.001), and total scleral area (SA_TOTAL) (96.14±34.38 vs. 56.91±14.97 mm²; P<0.001). The MPLDs at all angles showed significant differences in the 2 groups of eyes (P=0.008 at temporal 180°; P<0.001 at other angles). The greatest temporal-nasal asymmetry appeared at 75° apart from the midline in TAO eyes.

Conclusions

Our proposed system allowed automatic, comprehensive, and objective measurement of eyelid morphology by only using facial images, which has potential application prospects in TAO. Future work with a large sample of patients that contains different TAO subsets is warranted.

CROPro: a tool for automated cropping of prostate magnetic resonance images

Purpose

To bypass manual data preprocessing and optimize deep learning performance, we developed and evaluated CROPro, a tool to standardize automated cropping of prostate magnetic resonance (MR) images.

Approach

CROPro enables automatic cropping of MR images regardless of patient health status, image size, prostate volume, or pixel spacing. CROPro can crop foreground pixels from a region of interest (e.g., prostate) with different image sizes, pixel spacing, and sampling strategies. Performance was evaluated in the context of clinically significant prostate cancer (csPCa) classification. Transfer learning was used to train five convolutional neural network (CNN) and five vision transformer (ViT) models using different combinations of cropped image sizes ( 64 × 64 , 128 × 128 , and 256 × 256  pixels2), pixel spacing ( 0.2 × 0.2 , 0.3 × 0.3 , 0.4 × 0.4 , and 0.5 × 0.5    mm 2 ), and sampling strategies (center, random, and stride cropping) over the prostate. T2-weighted MR images ( N = 1475 ) from the online available PI-CAI challenge were used to train ( N = 1033 ), validate ( N = 221 ), and test ( N = 221 ) all models.

Results

Among CNNs, SqueezeNet with stride cropping (image size: 128 × 128 , pixel spacing: 0.2 × 0.2    mm 2 ) achieved the best classification performance ( 0.678 ± 0.006 ). Among ViTs, ViT-H/14 with random cropping (image size: 64 × 64 and pixel spacing: 0.5 × 0.5    mm 2 ) achieved the best performance ( 0.756 ± 0.009 ). Model performance depended on the cropped area, with optimal size generally larger with center cropping ( ∼ 40    cm 2 ) than random/stride cropping ( ∼ 10    cm 2 ).

Conclusion

We found that csPCa classification performance of CNNs and ViTs depends on the cropping settings. We demonstrated that CROPro is well suited to optimize these settings in a standardized manner, which could improve the overall performance of deep learning models.

Automated analysis framework for in vivo cardiac ablation therapy monitoring with optical coherence tomography

Radiofrequency ablation (RFA) is a minimally invasive procedure that is commonly used for the treatment of atrial fibrillation. However, it is associated with a significant risk of arrhythmia recurrence and complications owing to the lack of direct visualization of cardiac substrates and real-time feedback on ablation lesion transmurality. Within this manuscript, we present an automated deep learning framework for in vivo intracardiac optical coherence tomography (OCT) analysis of swine left atria. Our model can accurately identify cardiac substrates, monitor catheter-tissue contact stability, and assess lesion transmurality on both OCT intensity and polarization-sensitive OCT data. To the best of our knowledge, we have developed the first automatic framework for in vivo cardiac OCT analysis, which holds promise for real-time monitoring and guidance of cardiac RFA therapy..

Deep ensemble learning enables highly accurate classification of stored red blood cell morphology

Changes in red blood cell (RBC) morphology distribution have emerged as a quantitative biomarker for the degradation of RBC functional properties during hypothermic storage. Previously published automated methods for classifying the morphology of stored RBCs often had insufficient accuracy and relied on proprietary code and datasets, making them difficult to use in many research and clinical applications. Here we describe the development and validation of a highly accurate open-source RBC morphology classification pipeline based on ensemble deep learning (DL). The DL-enabled pipeline utilized adaptive thresholding or semantic segmentation for RBC identification, a deep ensemble of four convolutional neural networks (CNNs) to classify RBC morphology, and Kalman filtering with Hungarian assignment for tracking changes in the morphology of individual RBCs over time. The ensembled CNNs were trained and evaluated on thousands of individual RBCs from two open-access datasets previously collected to quantify the morphological heterogeneity and washing-induced shape recovery of stored RBCs. Confusion matrices and reliability diagrams demonstrated under-confidence of the constituent models and an accuracy of about 98% for the deep ensemble. Such a high accuracy allowed the CNN ensemble to uncover new insights over our previously published studies. Re-analysis of the datasets yielded much more accurate distributions of the effective diameters of stored RBCs at each stage of morphological degradation (discocyte: 7.821 ± 0.429 µm, echinocyte 1: 7.800 ± 0.581 µm, echinocyte 2: 7.304 ± 0.567 µm, echinocyte 3: 6.433 ± 0.490 µm, sphero-echinocyte: 5.963 ± 0.348 µm, spherocyte: 5.904 ± 0.292 µm, stomatocyte: 7.080 ± 0.522 µm). The effective diameter distributions were significantly different across all morphologies, with considerable effect sizes for non-neighboring classes. A combination of morphology classification with cell tracking enabled the discovery of a relatively rare and previously overlooked shape recovery of some sphero-echinocytes to early-stage echinocytes after washing with 1% human serum albumin solution. Finally, the datasets and code have been made freely available online to enable replication, further improvement, and adaptation of our work for other applications.

Training Deep Learning Models to Work on Multiple Devices by Cross-Domain Learning with No Additional Annotations

PURPOSE

To create an unsupervised cross-domain segmentation algorithm for segmenting intraretinal fluid and retinal layers on normal and pathologic macular OCT images from different manufacturers and camera devices.

DESIGN

We sought to use generative adversarial networks (GANs) to generalize a segmentation model trained on one OCT device to segment B-scans obtained from a different OCT device manufacturer in a fully unsupervised approach without labeled data from the latter manufacturer.

PARTICIPANTS

A total of 732 OCT B-scans from 4 different OCT devices (Heidelberg Spectralis, Topcon 1000, Maestro2, and Zeiss Plex Elite 9000).

METHODS

We developed an unsupervised GAN model, GANSeg, to segment 7 retinal layers and intraretinal fluid in Topcon 1000 OCT images (domain B) that had access only to labeled data on Heidelberg Spectralis images (domain A). GANSeg was unsupervised because it had access only to 110 Heidelberg labeled OCTs and 556 raw and unlabeled Topcon 1000 OCTs. To validate GANSeg segmentations, 3 masked graders manually segmented 60 OCTs from an external Topcon 1000 test dataset independently. To test the limits of GANSeg, graders also manually segmented 3 OCTs from Zeiss Plex Elite 9000 and Topcon Maestro2. A U-Net was trained on the same labeled Heidelberg images as baseline. The GANSeg repository with labeled annotations is at https://github.com/uw-biomedical-ml/ganseg.

MAIN OUTCOME MEASURES

Dice scores comparing segmentation results from GANSeg and the U-Net model with the manual segmented images.

RESULTS

Although GANSeg and U-Net achieved comparable Dice scores performance as human experts on the labeled Heidelberg test dataset, only GANSeg achieved comparable Dice scores with the best performance for the ganglion cell layer plus inner plexiform layer (90%; 95% confidence interval [CI], 68%-96%) and the worst performance for intraretinal fluid (58%; 95% CI, 18%-89%), which was statistically similar to human graders (79%; 95% CI, 43%-94%). GANSeg significantly outperformed the U-Net model. Moreover, GANSeg generalized to both Zeiss and Topcon Maestro2 swept-source OCT domains, which it had never encountered before.

CONCLUSIONS

GANSeg enables the transfer of supervised deep learning algorithms across OCT devices without labeled data, thereby greatly expanding the applicability of deep learning algorithms.

Grading of gliomas using transfer learning on MRI images

OBJECTIVE

Despite the critical role of Magnetic Resonance Imaging (MRI) in the diagnosis of brain tumours, there are still many pitfalls in the exact grading of them, in particular, gliomas. In this regard, it was aimed to examine the potential of Transfer Learning (TL) and Machine Learning (ML) algorithms in the accurate grading of gliomas on MRI images.

MATERIALS AND METHODS

Dataset has included four types of axial MRI images of glioma brain tumours with grades I-IV: T1-weighted, T2-weighted, FLAIR, and T1-weighted Contrast-Enhanced (T1-CE). Images were resized, normalized, and randomly split into training, validation, and test sets. ImageNet pre-trained Convolutional Neural Networks (CNNs) were utilized for feature extraction and classification, using Adam and SGD optimizers. Logistic Regression (LR) and Support Vector Machine (SVM) methods were also implemented for classification instead of Fully Connected (FC) layers taking advantage of features extracted by each CNN.

RESULTS

Evaluation metrics were computed to find the model with the best performance, and the highest overall accuracy of 99.38% was achieved for the model containing an SVM classifier and features extracted by pre-trained VGG-16.

DISCUSSION

It was demonstrated that developing Computer-aided Diagnosis (CAD) systems using pre-trained CNNs and classification algorithms is a functional approach to automatically specify the grade of glioma brain tumours in MRI images. Using these models is an excellent alternative to invasive methods and helps doctors diagnose more accurately before treatment.

A Novel Contrastive Self-Supervised Learning Framework for Solving Data Imbalance in Solder Joint Defect Detection

Poor chip solder joints can severely affect the quality of the finished printed circuit boards (PCBs). Due to the diversity of solder joint defects and the scarcity of anomaly data, it is a challenging task to automatically and accurately detect all types of solder joint defects in the production process in real time. To address this issue, we propose a flexible framework based on contrastive self-supervised learning (CSSL). In this framework, we first design several special data augmentation approaches to generate abundant synthetic, not good (sNG) data from the normal solder joint data. Then, we develop a data filter network to distill the highest quality data from sNG data. Based on the proposed CSSL framework, a high-accuracy classifier can be obtained even when the available training data are very limited. Ablation experiments verify that the proposed method can effectively improve the ability of the classifier to learn normal solder joint (OK) features. Through comparative experiments, the classifier trained with the help of the proposed method can achieve an accuracy of 99.14% on the test set, which is better than other competitive methods. In addition, its reasoning time is less than 6 ms per chip image, which is in favor of the real-time defect detection of chip solder joints.

GlottisNetV2: Temporal Glottal Midline Detection Using Deep Convolutional Neural Networks

High-speed videoendoscopy is a major tool for quantitative laryngology. Glottis segmentation and glottal midline detection are crucial for computing vocal fold-specific, quantitative parameters. However, fully automated solutions show limited clinical applicability. Especially unbiased glottal midline detection remains a challenging problem. We developed a multitask deep neural network for glottis segmentation and glottal midline detection. We used techniques from pose estimation to estimate the anterior and posterior points in endoscopy images. Neural networks were set up in TensorFlow/Keras and trained and evaluated with the BAGLS dataset. We found that a dual decoder deep neural network termed GlottisNetV2 outperforms the previously proposed GlottisNet in terms of MAPE on the test dataset (1.85% to 6.3%) while converging faster. Using various hyperparameter tunings, we allow fast and directed training. Using temporal variant data on an additional data set designed for this task, we can improve the median prediction accuracy from 2.1% to 1.76% when using 12 consecutive frames and additional temporal filtering. We found that temporal glottal midline detection using a dual decoder architecture together with keypoint estimation allows accurate midline prediction. We show that our proposed architecture allows stable and reliable glottal midline predictions ready for clinical use and analysis of symmetry measures.

Conditional Generative Adversarial Networks for Data Augmentation of a Neonatal Image Dataset

In today's neonatal intensive care units, monitoring vital signs such as heart rate and respiration is fundamental for neonatal care. However, the attached sensors and electrodes restrict movement and can cause medical-adhesive-related skin injuries due to the immature skin of preterm infants, which may lead to serious complications. Thus, unobtrusive camera-based monitoring techniques in combination with image processing algorithms based on deep learning have the potential to allow cable-free vital signs measurements. Since the accuracy of deep-learning-based methods depends on the amount of training data, proper validation of the algorithms is difficult due to the limited image data of neonates. In order to enlarge such datasets, this study investigates the application of a conditional generative adversarial network for data augmentation by using edge detection frames from neonates to create RGB images. Different edge detection algorithms were used to validate the input images' effect on the adversarial network's generator. The state-of-the-art network architecture Pix2PixHD was adapted, and several hyperparameters were optimized. The quality of the generated RGB images was evaluated using a Mechanical Turk-like multistage survey conducted by 30 volunteers and the FID score. In a fake-only stage, 23% of the images were categorized as real. A direct comparison of generated and real (manually augmented) images revealed that 28% of the fake data were evaluated as more realistic. An FID score of 103.82 was achieved. Therefore, the conducted study shows promising results for the training and application of conditional generative adversarial networks to augment highly limited neonatal image datasets.

Natural Language Processing Applications for Computer-Aided Diagnosis in Oncology

In the era of big data, text-based medical data, such as electronic health records (EHR) and electronic medical records (EMR), are growing rapidly. EHR and EMR are collected from patients to record their basic information, lab tests, vital signs, clinical notes, and reports. EHR and EMR contain the helpful information to assist oncologists in computer-aided diagnosis and decision making. However, it is time consuming for doctors to extract the valuable information they need and analyze the information from the EHR and EMR data. Recently, more and more research works have applied natural language processing (NLP) techniques, i.e., rule-based, machine learning-based, and deep learning-based techniques, on the EHR and EMR data for computer-aided diagnosis in oncology. The objective of this review is to narratively review the recent progress in the area of NLP applications for computer-aided diagnosis in oncology. Moreover, we intend to reduce the research gap between artificial intelligence (AI) experts and clinical specialists to design better NLP applications. We originally identified 295 articles from the three electronic databases: PubMed, Google Scholar, and ACL Anthology; then, we removed the duplicated papers and manually screened the irrelevant papers based on the content of the abstract; finally, we included a total of 23 articles after the screening process of the literature review. Furthermore, we provided an in-depth analysis and categorized these studies into seven cancer types: breast cancer, lung cancer, liver cancer, prostate cancer, pancreatic cancer, colorectal cancer, and brain tumors. Additionally, we identified the current limitations of NLP applications on supporting the clinical practices and we suggest some promising future research directions in this paper.

Computer-Aided Early Melanoma Brain-Tumor Detection Using Deep-Learning Approach

Brain tumors affect the normal functioning of the brain and if not treated in time these cancerous cells may affect the other tissues, blood vessels, and nerves surrounding these cells. Today, a large population worldwide is affected by the precarious disease of the brain tumor. Healthy tissues of the brain are suspected to be damaged because of tumors that become the most significant reason for a large number of deaths nowadays. Therefore, their early detection is necessary to prevent patients from unfortunate mishaps resulting in loss of lives. The manual detection of brain tumors is a challenging task due to discrepancies in appearance in terms of shape, size, nucleus, etc. As a result, an automatic system is required for the early detection of brain tumors. In this paper, the detection of tumors in brain cells is carried out using a deep convolutional neural network with stochastic gradient descent (SGD) optimization algorithm. The multi-classification of brain tumors is performed using the ResNet-50 model and evaluated on the public Kaggle brain-tumor dataset. The method achieved 99.82% and 99.5% training and testing accuracy, respectively. The experimental result indicates that the proposed model outperformed baseline methods, and provides a compelling reason to be applied to other diseases.

Ensembled EfficientNetB3 architecture for multi-class classification of tumours in MRI images

Healthcare informatics is one of the major concern domains in the processing of medical imaging for the diagnosis and treatment of brain tumours all over the world. Timely diagnosis of abnormal structures in brain tumours helps the clinical applications, medicines, doctors etc. in processing and analysing the medical imaging. The multi-class image classification of brain tumours faces challenges such as the scaling of large dataset, training of image datasets, efficiency, accuracy etc. EfficientNetB3 neural network scales the images in three dimensions resulting in improved accuracy. The novel neural network framework utilizes the optimization of an ensembled architecture of EfficientNetB3 with U-Net for MRI images which applies a semantic segmentation model for pre-trained backbone networks. The proposed neural model operates on a substantial network which will adapt the robustness by capturing the extraction of features in the U-Net encoder. The decoder will be enabling pixel-level localization at the definite precision level by an average ensemble of segmentation models. The ensembled pre-trained models will provide better training and prediction of abnormal structures in MRI images and thresholds for multi-classification of medical image visualization. The proposed model results in mean accuracy of 99.24 on the Kaggle dataset with 3064 images with a mean Dice score coefficient (DSC) of 0.9124 which is being compared with two state-of-art neural models.

Siam Deep Feature KCF Method and Experimental Study for Pedestrian Tracking

The tracking of a particular pedestrian is an important issue in computer vision to guarantee societal safety. Due to the limited computing performances of unmanned aerial vehicle (UAV) systems, the Correlation Filter (CF) algorithm has been widely used to perform the task of tracking. However, it has a fixed template size and cannot effectively solve the occlusion problem. Thus, a tracking-by-detection framework was designed in the current research. A lightweight YOLOv3-based (You Only Look Once version 3) mode which had Efficient Channel Attention (ECA) was integrated into the CF algorithm to provide deep features. In addition, a lightweight Siamese CNN with Cross Stage Partial (CSP) provided the representations of features learned from massive face images, allowing the target similarity in data association to be guaranteed. As a result, a Deep Feature Kernelized Correlation Filters method coupled with Siamese-CSP(Siam-DFKCF) was established to increase the tracking robustness. From the experimental results, it can be concluded that the anti-occlusion and re-tracking performance of the proposed method was increased. The tracking accuracy Distance Precision (DP) and Overlap Precision (OP) had been increased to 0.934 and 0.909 respectively in our test data.

Global Instance Tracking: Locating Target More Like Humans

Target tracking, the essential ability of the human visual system, has been simulated by computer vision tasks. However, existing trackers perform well in austere experimental environments but fail in challenges like occlusion and fast motion. The massive gap indicates that researches only measure tracking performance rather than intelligence. How to scientifically judge the intelligence level of trackers? Distinct from decision-making problems, lacking three requirements (a challenging task, a fair environment, and a scientific evaluation procedure) makes it strenuous to answer the question. In this article, we first propose the global instance tracking (GIT) task, which is supposed to search an arbitrary user-specified instance in a video without any assumptions about camera or motion consistency, to model the human visual tracking ability. Whereafter, we construct a high-quality and large-scale benchmark VideoCube to create a challenging environment. Finally, we design a scientific evaluation procedure using human capabilities as the baseline to judge tracking intelligence. Additionally, we provide an online platform with toolkit and an updated leaderboard. Although the experimental results indicate a definite gap between trackers and humans, we expect to take a step forward to generate authentic human-like trackers. The database, toolkit, evaluation server, and baseline results are available at http://videocube.aitestunion.com.

From 2D projections to the 3D rotation matrix: an attempt for finding a machine learning approach for the efficient evaluation of mechanical joining elements in X-ray computed tomography volume data

Destructive and predominantly manual procedures are commonly used in the automotive industry for the testing of mechanical joints, such as rivets or screws. Combining X-ray computed tomography (CT) and machine learning (ML) bears the potential of a non-destructive and largely automated methodology. Assuming the desired result is a comprehensible and documentable evaluation, three basic steps need to be automatized: First, a joint must be detected and identified as such in a CT scan of the joined parts. Second, the detected region containing the joint is rotated to a predefined orientation. Third, key measures in cross-sections from the newly oriented joint are dimensioned and documented. This work deals only with the second step, the rotation. On the one hand, we present a methodology for creating a well-curated data set for the contextual machine learning application. On the other, we evaluate its performance on the well-known ResNet50. More concretely, we investigate if it is possible for a deep convolutional neural network (CNN) to learn the respective rotation matrix from three volume projections that are perpendicular to each other. Two scenarios are investigated: In one scenario we assume that future data that is presented to the network has similar rivet demographics to historic data. We therefore do not employ hold-out sets for the network evaluation. In the other scenario we assume the opposite and therefore evaluating the networks performance with hold-out sets. We show that from a machine learning point of view, a CNN like ResNet50 is well able to learn this relationship with acceptable accuracy. In most cases the validation loss dropped below 0.1 after only a couple of epochs. In one particular case, we even reached both mean and median errors lower than 0.2 for approximately 80% of the entire test set of 1600 examples using our methodology. From an application point of view, however, these low test set errors should be treated with caution since small deviations from the intended rotation matrix can cause volume warping and translation. In another case, in which we used a hold-out set, only a fraction of the median errors were below 0.2.

Ambient intelligence-based monitoring of staff and patient activity in the intensive care unit

BACKGROUND

Caregiver workload in the ICU setting is difficult to numerically quantify. Ambient Intelligence utilises computer vision-guided neural networks to continuously monitor multiple datapoints in video feeds, has become increasingly efficient at automatically tracking various aspects of human movement.

OBJECTIVES

To assess the feasibility of using Ambient Intelligence to track and quantify allpatient and caregiver activity within a bedspace over the course of an ICU admission and also to establish patient specific factors, and environmental factors such as time ofday, that might contribute to an increased workload in ICU workers.

METHODS

5000 images were manually annotated and then used to train You Only LookOnce (YOLOv4), an open-source computer vision algorithm. Comparison of patientmotion and caregiver activity was then performed between these patients.

RESULTS

The algorithm was deployed on 14 patients comprising 1762800 framesof new, untrained data. There was a strong correlation between the number ofcaregivers in the room and the standardized movement of the patient (p < 0.0001) withmore caregivers associated with more movement. There was a significant difference incaregiver activity throughout the day (p < 0.05), HDU vs. ICU status (p < 0.05), delirious vs. non delirious patients (p < 0.05), and intubated vs. not intubated patients(p < 0.05). Caregiver activity was lowest between 0400 and 0800 (average .71 ± .026caregivers per hour) with statistically significant differences in activity compared to 0800-2400 (p < 0.05). Caregiver activity was highest between 1200 and 1600 (1.02 ± .031 caregivers per hour) with a statistically significant difference in activity comparedto activity from 1600 to 0800 (p < 0.05). The three most dominant predictors of workeractivity were patient motion (Standardized Dominance 78.6%), Mechanical Ventilation(Standardized Dominance 7.9%) and Delirium (Standardized Dominance 6.2%).

CONCLUSION

Ambient Intelligence could potentially be used to derive a single standardized metricthat could be applied to patients to illustrate their overall workload. This could be usedto predict workflow demands for better staff deployment, monitoring of caregiver workload, and potentially as a tool to predict burnout.

Machine Learning in Cardiovascular Imaging: A Scoping Review of Published Literature

Purpose of Review

In this study, we planned and carried out a scoping review of the literature to learn how machine learning (ML) has been investigated in cardiovascular imaging (CVI).

Recent Findings

During our search, we found numerous studies that developed or utilized existing ML models for segmentation, classification, object detection, generation, and regression applications involving cardiovascular imaging data. We first quantitatively investigated the different aspects of study characteristics, data handling, model development, and performance evaluation in all studies that were included in our review. We then supplemented these findings with a qualitative synthesis to highlight the common themes in the studied literature and provided recommendations to pave the way for upcoming research.

Summary

ML is a subfield of artificial intelligence (AI) that enables computers to learn human-like decision-making from data. Due to its novel applications, ML is gaining more and more attention from researchers in the healthcare industry. Cardiovascular imaging is an active area of research in medical imaging with lots of room for incorporating new technologies, like ML.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40134-022-00407-8.

Application of Artificial Intelligence to the Monitoring of Medication Adherence for Tuberculosis Treatment in Africa: Algorithm Development and Validation

Background

Artificial intelligence (AI) applications based on advanced deep learning methods in image recognition tasks can increase efficiency in the monitoring of medication adherence through automation. AI has sparsely been evaluated for the monitoring of medication adherence in clinical settings. However, AI has the potential to transform the way health care is delivered even in limited-resource settings such as Africa.

Objective

We aimed to pilot the development of a deep learning model for simple binary classification and confirmation of proper medication adherence to enhance efficiency in the use of video monitoring of patients in tuberculosis treatment.

Methods

We used a secondary data set of 861 video images of medication intake that were collected from consenting adult patients with tuberculosis in an institutional review board-approved study evaluating video-observed therapy in Uganda. The video images were processed through a series of steps to prepare them for use in a training model. First, we annotated videos using a specific protocol to eliminate those with poor quality. After the initial annotation step, 497 videos had sufficient quality for training the models. Among them, 405 were positive samples, whereas 92 were negative samples. With some preprocessing techniques, we obtained 160 frames with a size of 224 × 224 in each video. We used a deep learning framework that leveraged 4 convolutional neural networks models to extract visual features from the video frames and automatically perform binary classification of adherence or nonadherence. We evaluated the diagnostic properties of the different models using sensitivity, specificity, F1-score, and precision. The area under the curve (AUC) was used to assess the discriminative performance and the speed per video review as a metric for model efficiency. We conducted a 5-fold internal cross-validation to determine the diagnostic and discriminative performance of the models. We did not conduct external validation due to a lack of publicly available data sets with specific medication intake video frames.

Results

Diagnostic properties and discriminative performance from internal cross-validation were moderate to high in the binary classification tasks with 4 selected automated deep learning models. The sensitivity ranged from 92.8 to 95.8%, specificity from 43.5 to 55.4%, F1-score from 0.91 to 0.92, precision from 88% to 90.1%, and AUC from 0.78 to 0.85. The 3D ResNet model had the highest precision, AUC, and speed.

Conclusions

All 4 deep learning models showed comparable diagnostic properties and discriminative performance. The findings serve as a reasonable proof of concept to support the potential application of AI in the binary classification of video frames to predict medication adherence.

Development of an automated combined positive score prediction pipeline using artificial intelligence on multiplexed immunofluorescence images

Immunotherapy targeting immune checkpoint proteins, such as programmed cell death ligand 1 (PD-L1), has shown impressive outcomes in many clinical trials but only 20%-40% of patients benefit from it. Utilizing Combined Positive Score (CPS) to evaluate PD-L1 expression in tumour biopsies to identify patients with the highest likelihood of responsiveness to anti-PD-1/PD-L1 therapy has been approved by the Food and Drug Administration for several solid tumour types. Current CPS workflow requires a pathologist to manually score the two-colour PD-L1 chromogenic immunohistochemistry image. Multiplex immunofluorescence (mIF) imaging reveals the expression of an increased number of immune markers in tumour biopsies and has been used extensively in immunotherapy research. Recent rapid progress of Artificial Intelligence (AI)-based imaging analysis, particularly Deep Learning, provides cost effective and high-quality solutions to healthcare. In this article, we propose an imaging pipeline that utilizes three-colour mIF images (DAPI, PD-L1, and Pan-cytokeratin) as input and predicts the CPS using AI techniques. Our novel pipeline is composed of three modules employing algorithms of image processing, machine learning, and deep learning techniques. The first module of quality check (QC) detects and removes the image regions contaminated with sectioning and staining artefacts. The QC module ensures that only image regions free of the three common artefacts are used for downstream analysis. The second module of nuclear segmentation uses deep learning to segment and count nuclei in the DAPI images wherein our specialized method can accurately separate touching nuclei. The third module of cell phenotyping calculates CPS by identifying and counting PD-L1 positive cells and tumour cells. These modules are data-efficient and require only few manual annotations for training purposes. Using tumour biopsies from a clinical trial, we found that the CPS from the AI-based models shows a high Spearman correlation (78%, p = 0.003) to the pathologist-scored CPS.

Machine learning-assisted system using digital facial images to predict the clinical activity score in thyroid-associated orbitopathy

Although the clinical activity score (CAS) is a validated scoring system for identifying disease activity of thyroid-associated orbitopathy (TAO), it may produce differing results depending on the evaluator, and an experienced ophthalmologist is required for accurate evaluation. In this study, we developed a machine learning (ML)-assisted system to mimic an expert's CAS assessment using digital facial images and evaluated its accuracy for predicting the CAS and diagnosing active TAO (CAS ≥ 3). An ML-assisted system was designed to assess five CAS components related to inflammatory signs (redness of the eyelids, redness of the conjunctiva, swelling of the eyelids, inflammation of the caruncle and/or plica, and conjunctival edema) in patients' facial images and to predict the CAS by considering two components of subjective symptoms (spontaneous retrobulbar pain and pain on gaze). To train and test the system, 3,060 cropped images from 1020 digital facial images of TAO patients were used. The reference CAS for each image was scored by three ophthalmologists, each with > 15 years of clinical experience. We repeated the experiments for 30 randomly split training and test sets at a ratio of 8:2. The sensitivity and specificity of the ML-assisted system for diagnosing active TAO were 72.7% and 83.2% in the test set constructed from the entire dataset. For the test set constructed from the dataset with consistent results for the three ophthalmologists, the sensitivity and specificity for diagnosing active TAO were 88.1% and 86.9%. In the test sets from the entire dataset and from the dataset with consistent results, 40.0% and 49.9% of the predicted CAS values were the same as the reference CAS, respectively. The system predicted the CAS within 1 point of the reference CAS in 84.6% and 89.0% of cases when tested using the entire dataset and in the dataset with consistent results, respectively. An ML-assisted system estimated the clinical activity of TAO and detect inflammatory active TAO with reasonable accuracy. The accuracy could be improved further by obtaining more data. This ML-assisted system can help evaluate the disease activity consistently as well as accurately and enable the early diagnosis and timely treatment of active TAO.

The quest for the missing links in fatty liver genetics: Deep learning to the rescue!

Park, MacLean, et al. conduct an exome-wide association study of liver fat content in the Penn Medicine BioBank.¹ By leveraging machine learning-assisted analysis of clinical CT scans to quantify steatosis, they uncover previously undescribed liver fat-associated genetic variants.

Using AI and computer vision to analyze technical proficiency in robotic surgery

BACKGROUND

Intraoperative skills assessment is time-consuming and subjective; an efficient and objective computer vision-based approach for feedback is desired. In this work, we aim to design and validate an interpretable automated method to evaluate technical proficiency using colorectal robotic surgery videos with artificial intelligence.

METHODS

92 curated clips of peritoneal closure were characterized by both board-certified surgeons and a computer vision AI algorithm to compare the measures of surgical skill. For human ratings, six surgeons graded clips according to the GEARS assessment tool; for AI assessment, deep learning computer vision algorithms for surgical tool detection and tracking were developed and implemented.

RESULTS

For the GEARS category of efficiency, we observe a positive correlation between human expert ratings of technical efficiency and AI-determined total tool movement (r = - 0.72). Additionally, we show that more proficient surgeons perform closure with significantly less tool movement compared to less proficient surgeons (p < 0.001). For the GEARS category of bimanual dexterity, a positive correlation between expert ratings of bimanual dexterity and the AI model's calculated measure of bimanual movement based on simultaneous tool movement (r = 0.48) was also observed. On average, we also find that higher skill clips have significantly more simultaneous movement in both hands compared to lower skill clips (p < 0.001).

CONCLUSIONS

In this study, measurements of technical proficiency extracted from AI algorithms are shown to correlate with those given by expert surgeons. Although we target measurements of efficiency and bimanual dexterity, this work suggests that artificial intelligence through computer vision holds promise for efficiently standardizing grading of surgical technique, which may help in surgical skills training.

Towards successful implementation of artificial intelligence in skin cancer care: a qualitative study exploring the views of dermatologists and general practitioners

Recent studies show promising potential for artificial intelligence (AI) to assist healthcare providers (HCPs) in skin cancer care. The aim of this study is to explore the views of dermatologists and general practitioners (GPs) regarding the successful implementation of AI when assisting HCPs in skin cancer care. We performed a qualitative focus group study, consisting of six focus groups with 16 dermatologists and 17 GPs, varying in prior knowledge and experience with AI, gender, and age. An in-depth inductive thematic content analysis was deployed. Perceived benefits, barriers, and preconditions were identified as main themes. Dermatologists and GPs perceive substantial benefits of AI, particularly an improved health outcome and care pathway between primary and secondary care. Doubts about accuracy, risk of health inequalities, and fear of replacement were among the most stressed barriers. Essential preconditions included adequate algorithm content, sufficient usability, and accessibility of AI. In conclusion, dermatologists and GPs perceive significant benefits from implementing AI in skin cancer care. However, to successfully implement AI, key barriers need to be addressed. Efforts should focus on ensuring algorithm transparency, validation, accessibility for all skin types, and adequate regulation of algorithms. Simultaneously, improving knowledge about AI could reduce the fear of replacement.

Development and Verification of Postural Control Assessment Using Deep-Learning-Based Pose Estimators: Towards Clinical Applications

Occupational therapists evaluate various aspects of a client’s occupational performance. Among these, postural control is one of the fundamental skills that need assessment. Recently, several methods have been proposed to estimate postural control abilities using deep-learning-based approaches. Such techniques allow for the potential to provide automated, precise, fine-grained quantitative indices simply by evaluating videos of a client engaging in a postural control task. However, the clinical applicability of these assessment tools requires further investigation. In the current study, we compared three deep-learning-based pose estimators to assess their clinical applicability in terms of accuracy of pose estimations and processing speed. In addition, we verified which of the proposed quantitative indices for postural controls best reflected the clinical evaluations of occupational therapists. A framework using deep-learning techniques broadens the possibility of quantifying clients’ postural control in a more fine-grained way compared with conventional coarse indices, which can lead to improved occupational therapy practice.

Co-Occurrence Fingerprint Data-Based Heterogeneous Transfer Learning Framework for Indoor Positioning

Distribution discrepancy is an intrinsic challenge in existing fingerprint-based indoor positioning system(s) (FIPS) due to real-time environmental variations; thus, the positioning model needs to be reconstructed frequently based on newly collected training data. However, it is expensive or impossible to collect adequate training samples to reconstruct the fingerprint database. Fortunately, transfer learning has proven to be an effective solution to mitigate the distribution discrepancy, enabling us to update the positioning model using newly collected training data in real time. However, in practical applications, traditional transfer learning algorithms no longer act well to feature space heterogeneity caused by different types or holding postures of fingerprint collection devices (such as smartphones). Moreover, current heterogeneous transfer methods typically require enough accurately labeled samples in the target domain, which is practically expensive and even unavailable. Aiming to solve these problems, a heterogeneous transfer learning framework based on co-occurrence data (HTL-CD) is proposed for FIPS, which can realize higher positioning accuracy and robustness against environmental changes without reconstructing the fingerprint database repeatedly. Specifically, the source domain samples are mapped into the feature space in the target domain, then the marginal and conditional distributions of the source and target samples are aligned in order to minimize the distribution divergence caused by collection device heterogeneity and environmental changes. Moreover, the utilized co-occurrence fingerprint data enables us to calculate correlation coefficients between heterogeneous samples without accurately labeled target samples. Furthermore, by resorting to the adopted correlation restriction mechanism, more valuable knowledge will be transferred to the target domain if the source samples are related to the target ones, which remarkably relieves the "negative transfer" issue. Real-world experimental performance implies that, even without accurately labeled samples in the target domain, the proposed HTL-CD can obtain at least 17.15% smaller average localization errors (ALEs) than existing transfer learning-based positioning methods, which further validates the effectiveness and superiority of our algorithm.

Smartphone video nystagmography using convolutional neural networks: ConVNG

Background

Eye movement abnormalities are commonplace in neurological disorders. However, unaided eye movement assessments lack granularity. Although videooculography (VOG) improves diagnostic accuracy, resource intensiveness precludes its broad use. To bridge this care gap, we here validate a framework for smartphone video-based nystagmography capitalizing on recent computer vision advances.

Methods

A convolutional neural network was fine-tuned for pupil tracking using > 550 annotated frames: ConVNG. In a cross-sectional approach, slow-phase velocity of optokinetic nystagmus was calculated in 10 subjects using ConVNG and VOG. Equivalence of accuracy and precision was assessed using the “two one-sample t-test” (TOST) and Bayesian interval-null approaches. ConVNG was systematically compared to OpenFace and MediaPipe as computer vision (CV) benchmarks for gaze estimation.

Results

ConVNG tracking accuracy reached 9–15% of an average pupil diameter. In a fully independent clinical video dataset, ConVNG robustly detected pupil keypoints (median prediction confidence 0.85). SPV measurement accuracy was equivalent to VOG (TOST p < 0.017; Bayes factors (BF) > 24). ConVNG, but not MediaPipe, achieved equivalence to VOG in all SPV calculations. Median precision was 0.30°/s for ConVNG, 0.7°/s for MediaPipe and 0.12°/s for VOG. ConVNG precision was significantly higher than MediaPipe in vertical planes, but both algorithms’ precision was inferior to VOG.

Conclusions

ConVNG enables offline smartphone video nystagmography with an accuracy comparable to VOG and significantly higher precision than MediaPipe, a benchmark computer vision application for gaze estimation. This serves as a blueprint for highly accessible tools with potential to accelerate progress toward precise and personalized Medicine.

The Role of Natural Language Processing during the COVID-19 Pandemic: Health Applications, Opportunities, and Challenges

The COVID-19 pandemic is the most devastating public health crisis in at least a century and has affected the lives of billions of people worldwide in unprecedented ways. Compared to pandemics of this scale in the past, societies are now equipped with advanced technologies that can mitigate the impacts of pandemics if utilized appropriately. However, opportunities are currently not fully utilized, particularly at the intersection of data science and health. Health-related big data and technological advances have the potential to significantly aid the fight against such pandemics, including the current pandemic's ongoing and long-term impacts. Specifically, the field of natural language processing (NLP) has enormous potential at a time when vast amounts of text-based data are continuously generated from a multitude of sources, such as health/hospital systems, published medical literature, and social media. Effectively mitigating the impacts of the pandemic requires tackling challenges associated with the application and deployment of NLP systems. In this paper, we review the applications of NLP to address diverse aspects of the COVID-19 pandemic. We outline key NLP-related advances on a chosen set of topics reported in the literature and discuss the opportunities and challenges associated with applying NLP during the current pandemic and future ones. These opportunities and challenges can guide future research aimed at improving the current health and social response systems and pandemic preparedness.