Evaluating robotic-assisted partial nephrectomy surgeons with fully convolutional segmentation and multi-task attention networks

We use machine learning to evaluate surgical skill from videos during the tumor resection and renography steps of a robotic assisted partial nephrectomy (RAPN). This expands previous work using synthetic tissue to include actual surgeries. We investigate cascaded neural networks for predicting surgical proficiency scores (OSATS and GEARS) from RAPN videos recorded from the DaVinci system. The semantic segmentation task generates a mask and tracks the various surgical instruments. The movements from the instruments found via semantic segmentation are processed by a scoring network that regresses (predicts) GEARS and OSATS scoring for each subcategory. Overall, the model performs well for many subcategories such as force sensitivity and knowledge of instruments of GEARS and OSATS scoring, but can suffer from false positives and negatives that would not be expected of human raters. This is mainly attributed to limited training data variability and sparsity.

Towards automated molecular detection through simulated generation of CMOS-based rotational spectroscopy

The use of CMOS sensors for rotational spectroscopy is a promising, but challenging avenue for low-cost gas sensing and molecular identification. A main challenge in this approach is that practical CMOS spectroscopy samples contain various different noise sources that reduce the effectiveness of matching techniques for molecular identification with rotational spectroscopy. To help solve this challenge, we develop a software application tool that can demonstrate the feasibility and reliability of detection with CMOS sensor samples. Specifically, the tool characterizes the types of noise in CMOS sample collection and synthesizes spectroscopy files based upon existing databases of rotational spectroscopy samples gathered from other sensors. We use the software to create a large database of plausible CMOS-generated sample files of gases. This dataset is used to help evaluate spectral matching algorithms used in gas sensing and molecular identification applications. We evaluate these traditional methods on the synthesized dataset and discuss how peak finding and spectral matching algorithms can be altered to accommodate the noise sources present in CMOS sample collection.

LAST: Latent Space-Assisted Adaptive Sampling for Protein Trajectories

Molecular dynamics (MD) simulation is widely used to study protein conformations and dynamics. However, conventional simulation suffers from being trapped in some local energy minima that are hard to escape. Thus, most of the computational time is spent sampling in the already visited regions. This leads to an inefficient sampling process and further hinders the exploration of protein movements in affordable simulation time. The advancement of deep learning provides new opportunities for protein sampling. Variational autoencoders are a class of deep learning models to learn a low-dimensional representation (referred to as the latent space) that can capture the key features of the input data. Based on this characteristic, we proposed a new adaptive sampling method, latent space-assisted adaptive sampling for protein trajectories (LAST), to accelerate the exploration of protein conformational space. This method comprises cycles of (i) variational autoencoder training, (ii) seed structure selection on the latent space, and (iii) conformational sampling through additional MD simulations. The proposed approach is validated through the sampling of four structures of two protein systems: two metastable states of Escherichia coli adenosine kinase (ADK) and two native states of Vivid (VVD). In all four conformations, seed structures were shown to lie on the boundary of conformation distributions. Moreover, large conformational changes were observed in a shorter simulation time when compared with structural dissimilarity sampling (SDS) and conventional MD (cMD) simulations in both systems. In metastable ADK simulations, LAST explored two transition paths toward two stable states, while SDS explored only one and cMD neither. In VVD light state simulations, LAST was three times faster than cMD simulation with a similar conformational space. Overall, LAST is comparable to SDS and is a promising tool in adaptive sampling. The LAST method is publicly available at https://github.com/smu-tao-group/LAST to facilitate related research.

Smartphone camera oximetry in an induced hypoxemia study

Hypoxemia, a medical condition that occurs when the blood is not carrying enough oxygen to adequately supply the tissues, is a leading indicator for dangerous complications of respiratory diseases like asthma, COPD, and COVID-19. While purpose-built pulse oximeters can provide accurate blood-oxygen saturation (SpO₂) readings that allow for diagnosis of hypoxemia, enabling this capability in unmodified smartphone cameras via a software update could give more people access to important information about their health. Towards this goal, we performed the first clinical development validation on a smartphone camera-based SpO₂ sensing system using a varied fraction of inspired oxygen (FiO₂) protocol, creating a clinically relevant validation dataset for solely smartphone-based contact PPG methods on a wider range of SpO₂ values (70–100%) than prior studies (85–100%). We built a deep learning model using this data to demonstrate an overall MAE = 5.00% SpO₂ while identifying positive cases of low SpO₂ < 90% with 81% sensitivity and 79% specificity. We also provide the data in open-source format, so that others may build on this work.

Explore Protein Conformational Space With Variational Autoencoder

Molecular dynamics (MD) simulations have been actively used in the study of protein structure and function. However, extensive sampling in the protein conformational space requires large computational resources and takes a prohibitive amount of time. In this study, we demonstrated that variational autoencoders (VAEs), a type of deep learning model, can be employed to explore the conformational space of a protein through MD simulations. VAEs are shown to be superior to autoencoders (AEs) through a benchmark study, with low deviation between the training and decoded conformations. Moreover, we show that the learned latent space in the VAE can be used to generate unsampled protein conformations. Additional simulations starting from these generated conformations accelerated the sampling process and explored hidden spaces in the conformational landscape.

Evaluating robotic-assisted surgery training videos with multi-task convolutional neural networks

We seek to understand if an automated algorithm can replace human scoring of surgical trainees performing the urethrovesical anastomosis in radical prostatectomy with synthetic tissue. Specifically, we investigate neural networks for predicting the surgical proficiency score (GEARS score) from video clips. We evaluate videos of surgeons performing the urethral anastomosis using synthetic tissue. The algorithm tracks surgical instrument locations from video, saving the positions of key points on the instruments over time. These positional features are used to train a multi-task convolutional network to infer each sub-category of the GEARS score to determine the proficiency level of trainees. Experimental results demonstrate that the proposed method achieves good performance with scores matching manual inspection in 86.1% of all GEARS sub-categories. Furthermore, the model can detect the difference between proficiency (novice to expert) in 83.3% of videos. Evaluation of GEARS sub-categories with artificial neural networks is possible for novice and intermediate surgeons, but additional research is needed to understand if expert surgeons can be evaluated with a similar automated system.

Generative adversarial networks for transition state geometry prediction

This work introduces a novel application of generative adversarial networks (GANs) for the prediction of starting geometries in transition state (TS) searches based on the geometries of reactants and products. The multi-dimensional potential energy space of a chemical reaction often complicates the location of a starting TS geometry, leading to the correct TS combining reactants and products in question. The proposed TS-GAN efficiently maps the space between reactants and products and generates reliable TS guess geometries, and it can be easily combined with any quantum chemical software package performing geometry optimizations. The TS-GAN was trained and applied to generate TS guess structures for typical chemical reactions, such as hydrogen migration, isomerization, and transition metal-catalyzed reactions. The performance of the TS-GAN was directly compared to that of classical approaches, proving its high accuracy and efficiency. The current TS-GAN can be extended to any dataset that contains sufficient chemical reactions for training. The software is freely available for training, experimentation, and prediction at https://github.com/ekraka/TS-GAN.

Deep Learning-Based Ligand Design Using Shared Latent Implicit Fingerprints from Collaborative Filtering

In their previous work, Srinivas et al. [ J. Cheminf. 2018, 10, 56] have shown that implicit fingerprints capture ligands and proteins in a shared latent space, typically for the purposes of virtual screening with collaborative filtering models applied on known bioactivity data. In this work, we extend these implicit fingerprints/descriptors using deep learning techniques to translate latent descriptors into discrete representations of molecules (SMILES), without explicitly optimizing for chemical properties. This allows the design of new compounds based upon the latent representation of nearby proteins, thereby encoding druglike properties including binding affinities to known proteins. The implicit descriptor method does not require any fingerprint similarity search, which makes the method free of any bias arising from the empirical nature of the fingerprint models [Srinivas, R.; J. Cheminf. 2018, 10, 56]. We evaluate the properties of the potentially novel drugs generated by our approach using physical properties of druglike molecules and chemical complexity. Additionally, we analyze the reliability of the biological activity of the new compounds generated using this method by employing models of protein-ligand interaction, which assists in assessing the potential binding affinity of the designed compounds. We find that the generated compounds exhibit properties of chemically feasible compounds and are predicted to be excellent binders to known proteins. Furthermore, we also analyze the diversity of compounds created using the Tanimoto distance and conclude that there is a wide diversity in the generated compounds.

Swapped face detection using deep learning and subjective assessment

The tremendous success of deep learning for imaging applications has resulted in numerous beneficial advances. Unfortunately, this success has also been a catalyst for malicious uses such as photo-realistic face swapping of parties without consent. In this study, we use deep transfer learning for face swapping detection, showing true positive rates greater than 96% with very few false alarms. Distinguished from existing methods that only provide detection accuracy, we also provide uncertainty for each prediction, which is critical for trust in the deployment of such detection systems. Moreover, we provide a comparison to human subjects. To capture human recognition performance, we build a website to collect pairwise comparisons of images from human subjects. Based on these comparisons, we infer a consensus ranking from the image perceived as most real to the image perceived as most fake. Overall, the results show the effectiveness of our method. As part of this study, we create a novel dataset that is, to the best of our knowledge, the largest swapped face dataset created using still images. This dataset will be available for academic research use per request. Our goal of this study is to inspire more research in the field of image forensics through the creation of a dataset and initial analysis.

Measuring Oxygen Saturation With Smartphone Cameras Using Convolutional Neural Networks

Arterial oxygen saturation ([Formula: see text]) is an indicator of how much oxygen is carried by hemoglobin in the blood. Having enough oxygen is vital for the functioning of cells in the human body. Measurement of [Formula: see text] is typically estimated with a pulse oximeter, but recent works have investigated how smartphone cameras can be used to infer [Formula: see text]. In this paper, we propose methods for the measurement of [Formula: see text] with a smartphone using convolutional neural networks and preprocessing steps to better guard against motion artifacts. To evaluate this methodology, we conducted a breath-holding study involving 39 participants. We compare the results using two different mobile phones. We compare our model with the ratio-of-ratios model that is widely used in pulse oximeter applications, showing that our system has significantly lower mean absolute error (2.02%) than a medical pulse oximeter.

Implicit-descriptor ligand-based virtual screening by means of collaborative filtering

Current ligand-based machine learning methods in virtual screening rely heavily on molecular fingerprinting for preprocessing, i.e., explicit description of ligands’ structural and physicochemical properties in a vectorized form. Of particular importance to current methods are the extent to which molecular fingerprints describe a particular ligand and what metric sufficiently captures similarity among ligands. In this work, we propose and evaluate methods that do not require explicit feature vectorization through fingerprinting, but, instead, provide implicit descriptors based only on other known assays. Our methods are based upon well known collaborative filtering algorithms used in recommendation systems. Our implicit descriptor method does not require any fingerprint similarity search, which makes the method free of the bias arising from the empirical nature of the fingerprint models. We show that implicit methods significantly outperform traditional machine learning methods, and the main strengths of implicit methods are their resilience to target-ligand sparsity and high potential for spotting promiscuous ligands.

Use of a Smartphone App to Assess Neonatal Jaundice

BACKGROUND

The assessment of jaundice in outpatient neonates is problematic. Visual assessment is inaccurate, and more exact methodologies are cumbersome and/or expensive. Our goal in this study was to assess the accuracy of a technology based on the analysis of digital images of newborns obtained using a smartphone application called BiliCam.

METHODS

Paired BiliCam images and total serum bilirubin (TSB) levels were obtained in a diverse sample of newborns (<7 days old) at 7 sites across the United States. By using specialized software, data on color values in the images ("features") were extracted. Machine learning and regression analysis techniques were used to identify features for inclusion in models to predict an estimated bilirubin level for each newborn. The correlation between estimated bilirubin levels and TSB levels was calculated. In addition, the sensitivity and specificity of the estimated bilirubin levels in identifying newborns with high TSB levels were calculated by using 2 recommended decision rules for jaundice screening.

RESULTS

Estimated bilirubin levels were calculated and compared with TSB levels in a diverse sample of 530 newborns (20.8% African American, 26.3% Hispanic, and 21.2% Asian American). The overall correlation was 0.91, and correlations among white, African American, Hispanic, and Asian American newborns were 0.92, 0.90, 0.91, and 0.88, respectively. The sensitivities of BiliCam in identifying newborns with high TSB levels were 84.6% and 100%, respectively, by using 2 decision rules; specificities were 75.1% and 76.4%, respectively.

CONCLUSIONS

BiliCam provided accurate estimates of TSB values, demonstrating that an inexpensive technology that uses commodity smartphones could be used to effectively screen newborns for jaundice.