Automatic segmentation and classification of Papanicolaou-stained cells and dataset for oral cancer detection

BACKGROUND AND OBJECTIVE

Papanicolaou staining has been successfully used to assist early detection of cervix cancer for several decades. We postulate that this staining technique can also be used for assisting early detection of oral cancer, which is responsible for about 300,000 deaths every year. The rational for such claim includes two key observations: (i) nuclear atypia, i.e., changes in volume, shape, and staining properties of the cell nuclei can be linked to rapid cell proliferation and genetic instability; and (ii) Papanicolaou staining allows one to reliably segment cells' nuclei and cytoplasms. While Papanicolaou staining is an attractive tool due to its low cost, its interpretation requires a trained pathologist. Our goal is to automate the segmentation and classification of morphological features needed to evaluate the use of Papanicolaou staining for early detection of mouth cancer.

METHODS

We built a convolutional neural network (CNN) for automatic segmentation and classification of cells in Papanicolaou-stained images. Our CNN was trained and evaluated on a new image dataset of cells from oral mucosa consisting of 1,563 Full HD images from 52 patients, annotated by specialists. The effectiveness of our model was evaluated against a group of experts. Its robustness was also demonstrated on five public datasets of cervical images captured with different microscopes and cameras, and having different resolutions, colors, background intensities, and noise levels.

RESULTS

Our CNN model achieved expert-level performance in a comparison with a group of three human experts on a set of 400 Papanicolaou-stained images of the oral mucosa from 20 patients. The results of this experiment exhibited high Interclass Correlation Coefficient (ICC) values. Despite being trained on images from the oral mucosa, it produced high-quality segmentation and plausible classification for five public datasets of cervical cells. Our Papanicolaou-stained image dataset is the most diverse publicly available image dataset for the oral mucosa in terms of number of patients.

CONCLUSION

Our solution provides the means for exploring the potential of Papanicolaou-staining as a powerful and inexpensive tool for early detection of oral cancer. We are currently using our system to detect suspicious cells and cell clusters in oral mucosa slide images. Our trained model, code, and dataset are available and can help practitioners and stimulate research in early oral cancer detection.

Real-Time CNN Training and Compression for Neural-Enhanced Adaptive Live Streaming

We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

Recent advances in artificial intelligent strategies for tissue engineering and regenerative medicine

BACKGROUND

Tissue engineering and regenerative medicine (TERM) aim to repair or replace damaged or lost tissues or organs due to accidents, diseases, or aging, by applying different sciences. For this purpose, an essential part of TERM is the designing, manufacturing, and evaluating of scaffolds, cells, tissues, and organs. Artificial intelligence (AI) or the intelligence of machines or software can be effective in all areas where computers play a role.

METHODS

The "artificial intelligence," "machine learning," "tissue engineering," "clinical evaluation," and "scaffold" keywords used for searching in various databases and articles published from 2000 to 2024 were evaluated.

RESULTS

The combination of tissue engineering and AI has created a new generation of technological advancement in the biomedical industry. Experience in TERM has been refined using advanced design and manufacturing techniques. Advances in AI, particularly deep learning, offer an opportunity to improve scientific understanding and clinical outcomes in TERM.

CONCLUSION

The findings of this research show the high potential of AI, machine learning, and robots in the selection, design, and fabrication of scaffolds, cells, tissues, or organs, and their analysis, characterization, and evaluation after their implantation. AI can be a tool to accelerate the introduction of tissue engineering products to the bedside.

HIGHLIGHTS

The capabilities of artificial intelligence (AI) can be used in different ways in all the different stages of TERM and not only solve the existing limitations, but also accelerate the processes, increase efficiency and precision, reduce costs, and complications after transplantation. ML predicts which technologies have the most efficient and easiest path to enter the market and clinic. The use of AI along with these imaging techniques can lead to the improvement of diagnostic information, the reduction of operator errors when reading images, and the improvement of image analysis (such as classification, localization, regression, and segmentation).

Reusable Architecture Growth for Continual Stereo Matching

The remarkable performance of recent stereo depth estimation models benefits from the successful use of convolutional neural networks to regress dense disparity. Akin to most tasks, this needs gathering training data that covers a number of heterogeneous scenes at deployment time. However, training samples are typically acquired continuously in practical applications, making the capability to learn new scenes continually even more crucial. For this purpose, we propose to perform continual stereo matching where a model is tasked to 1) continually learn new scenes, 2) overcome forgetting previously learned scenes, and 3) continuously predict disparities at inference. We achieve this goal by introducing a Reusable Architecture Growth (RAG) framework. RAG leverages task-specific neural unit search and architecture growth to learn new scenes continually in both supervised and self-supervised manners. It can maintain high reusability during growth by reusing previous units while obtaining good performance. Additionally, we present a Scene Router module to adaptively select the scene-specific architecture path at inference. Comprehensive experiments on numerous datasets show that our framework performs impressively in various weather, road, and city circumstances and surpasses the state-of-the-art methods in more challenging cross-dataset settings. Further experiments also demonstrate the adaptability of our method to unseen scenes, which can facilitate end-to-end stereo architecture learning and practical deployment.

Significance of AI-assisted techniques for epiphyte plant monitoring and identification from drone images

Monitoring forest canopies is vital for ecological studies, particularly for assessing epiphytes in rain forest ecosystems. Traditional methods for studying epiphytes, such as climbing trees and building observation structures, are labor, cost intensive and risky. Unmanned Aerial Vehicles (UAVs) have emerged as a valuable tool in this domain, offering botanists a safer and more cost-effective means to collect data. This study leverages AI-assisted techniques to enhance the identification and mapping of epiphytes using UAV imagery. The primary objective of this research is to evaluate the effectiveness of AI-assisted methods compared to traditional approaches in segmenting/identifying epiphytes from UAV images collected in a reserve forest in Costa Rica. Specifically, the study investigates whether Deep Learning (DL) models can accurately identify epiphytes during complex backgrounds, even with a limited dataset of varying image quality. Systematically, this study compares three traditional image segmentation methods Auto Cluster, Watershed, and Level Set with two DL-based segmentation networks: the UNet and the Vision Transformer-based TransUNet. Results obtained from this study indicate that traditional methods struggle with the complexity of vegetation backgrounds and variability in target characteristics. Epiphyte identification results were quantitatively evaluated using the Jaccard score. Among traditional methods, Watershed scored 0.10, Auto Cluster 0.13, and Level Set failed to identify the target. In contrast, AI-assisted models performed better, with UNet scoring 0.60 and TransUNet 0.65. These results highlight the potential of DL approaches to improve the accuracy and efficiency of epiphyte identification and mapping, advancing ecological research and conservation.

Bi-Directional Ensemble Feature Reconstruction Network for Few-Shot Fine-Grained Classification

The main challenge for fine-grained few-shot image classification is to learn feature representations with higher inter-class and lower intra-class variations, with a mere few labelled samples. Conventional few-shot learning methods however cannot be naively adopted for this fine-grained setting - a quick pilot study reveals that they in fact push for the opposite (i.e., lower inter-class variations and higher intra-class variations). To alleviate this problem, prior works predominately use a support set to reconstruct the query image and then utilize metric learning to determine its category. Upon careful inspection, we further reveal that such unidirectional reconstruction methods only help to increase inter-class variations and are not effective in tackling intra-class variations. In this paper, we introduce a bi-reconstruction mechanism that can simultaneously accommodate for inter-class and intra-class variations. In addition to using the support set to reconstruct the query set for increasing inter-class variations, we further use the query set to reconstruct the support set for reducing intra-class variations. This design effectively helps the model to explore more subtle and discriminative features which is key for the fine-grained problem in hand. Furthermore, we also construct a self-reconstruction module to work alongside the bi-directional module to make the features even more discriminative. We introduce the snapshot ensemble method in the episodic learning strategy - a simple trick to further improve model performance without increasing training costs. Experimental results on three widely used fine-grained image classification datasets, as well as general and cross-domain few-shot image datasets, consistently show considerable improvements compared with other methods.

Artificial intelligence automated surgical phases recognition in intraoperative videos of laparoscopic pancreatoduodenectomy

BACKGROUND

Laparoscopic pancreatoduodenectomy (LPD) is one of the most challenging operations and has a long learning curve. Artificial intelligence (AI) automated surgical phase recognition in intraoperative videos has many potential applications in surgical education, helping shorten the learning curve, but no study has made this breakthrough in LPD. Herein, we aimed to build AI models to recognize the surgical phase in LPD and explore the performance characteristics of AI models.

METHODS

Among 69 LPD videos from a single surgical team, we used 42 in the building group to establish the models and used the remaining 27 videos in the analysis group to assess the models' performance characteristics. We annotated 13 surgical phases of LPD, including 4 key phases and 9 necessary phases. Two minimal invasive pancreatic surgeons annotated all the videos. We built two AI models for the key phase and necessary phase recognition, based on convolutional neural networks. The overall performance of the AI models was determined mainly by mean average precision (mAP).

RESULTS

Overall mAPs of the AI models in the test set of the building group were 89.7% and 84.7% for key phases and necessary phases, respectively. In the 27-video analysis group, overall mAPs were 86.8% and 71.2%, with maximum mAPs of 98.1% and 93.9%. We found commonalities between the error of model recognition and the differences of surgeon annotation, and the AI model exhibited bad performance in cases with anatomic variation or lesion involvement with adjacent organs.

CONCLUSIONS

AI automated surgical phase recognition can be achieved in LPD, with outstanding performance in selective cases. This breakthrough may be the first step toward AI- and video-based surgical education in more complex surgeries.

Active Dynamic Weighting for multi-domain adaptation

Multi-source unsupervised domain adaptation aims to transfer knowledge from multiple labeled source domains to an unlabeled target domain. Existing methods either seek a mixture of distributions across various domains or combine multiple single-source models for weighted fusion in the decision process, with little insight into the distributional discrepancy between different source domains and the target domain. Considering the discrepancies in global and local feature distributions between different domains and the complexity of obtaining category boundaries across domains, this paper proposes a novel Active Dynamic Weighting (ADW) for multi-source domain adaptation. Specifically, to effectively utilize the locally advantageous features in the source domains, ADW designs a multi-source dynamic adjustment mechanism during the training process to dynamically control the degree of feature alignment between each source and target domain in the training batch. In addition, to ensure the cross-domain categories can be distinguished, ADW devises a dynamic boundary loss to guide the model to focus on the hard samples near the decision boundary, which enhances the clarity of the decision boundary and improves the model's classification ability. Meanwhile, ADW applies active learning to multi-source unsupervised domain adaptation for the first time, guided by dynamic boundary loss, proposes an efficient importance sampling strategy to select target domain hard samples to annotate at a minimal annotation budget, integrates it into the training process, and further refines the domain alignment at the category level. Experiments on various benchmark datasets consistently demonstrate the superiority of our method.

Energy-efficient dynamic sensor time series classification for edge health devices

BACKGROUND AND OBJECTIVE

Time series data plays a crucial role in the realm of the Internet of Things Medical (IoMT). Through machine learning (ML) algorithms, online time series classification in IoMT systems enables reliable real-time disease detection. Deploying ML algorithms on edge health devices can reduce latency and safeguard patients' privacy. However, the limited computational resources of these devices underscore the need for more energy-efficient algorithms. Furthermore, online time series classification inevitably faces the challenges of concept drift (CD) and catastrophic forgetting (CF). To address these challenges, this study proposes an energy-efficient Online Time series classification algorithm that can solve CF and CD for health devices, called OTCD.

METHODS

OTCD first detects the appearance of concept drift and performs prototype updates to mitigate its impact. Afterward, it standardizes the potential space distribution and selectively preserves key training parameters to address CF. This approach reduces the required memory and enhances energy efficiency. To evaluate the performance of the proposed model in real-time health monitoring tasks, we utilize electrocardiogram (ECG) and photoplethysmogram (PPG) data. By adopting various feature extractors, three arrhythmia classification models are compared. To assess the energy efficiency of OTCD, we conduct runtime tests on each dataset. Additionally, the OTCD is compared with state-of-the-art (SOTA) dynamic time series classification models for performance evaluation.

RESULTS

The OTCD algorithm outperforms existing SOTA time series classification algorithms in IoMT. In particular, OTCD is on average 2.77% to 14.74% more accurate than other models on the MIT-BIH arrhythmia dataset. Additionally, it consumes low memory (1 KB) and performs computations at a rate of 0.004 GFLOPs per second, leading to energy savings and high time efficiency.

CONCLUSION

Our proposed algorithm, OTCD, enables efficient real-time classification of medical time series on edge health devices. Experimental results demonstrate its significant competitiveness, offering promising prospects for safe and reliable healthcare.

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition

Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Other than long-tailed problems, ProCo can be directly applied to semi-supervised learning by generating pseudo-labels for unlabeled data, which can subsequently be utilized to estimate the distribution of the samples inversely. Theoretically, we analyze the error bound of ProCo. Empirically, extensive experimental results on supervised/semi-supervised visual recognition and object detection tasks demonstrate that ProCo consistently outperforms existing methods across various datasets.

Robust Shape Fitting for 3D Scene Abstraction

Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

DualFluidNet: An attention-based dual-pipeline network for fluid simulation

Fluid motion can be considered as a point cloud transformation when using the SPH method. Compared to traditional numerical analysis methods, using machine learning techniques to learn physics simulations can achieve near-accurate results, while significantly increasing efficiency. In this paper, we propose an innovative approach for 3D fluid simulations utilizing an Attention-based Dual-pipeline Network, which employs a dual-pipeline architecture, seamlessly integrated with an Attention-based Feature Fusion Module. Unlike previous methods, which often make difficult trade-offs between global fluid control and physical law constraints, we find a way to achieve a better balance between these two crucial aspects with a well-designed dual-pipeline approach. Additionally, we design a Type-aware Input Module to adaptively recognize particles of different types and perform feature fusion afterward, such that fluid-solid coupling issues can be better dealt with. Furthermore, we propose a new dataset, Tank3D, to further explore the network's ability to handle more complicated scenes. The experiments demonstrate that our approach not only attains a quantitative enhancement in various metrics, surpassing the state-of-the-art methods, but also signifies a qualitative leap in neural network-based simulation by faithfully adhering to the physical laws. Code and video demonstrations are available at https://github.com/chenyu-xjtu/DualFluidNet.

Unpacking the Gap Box Against Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) improves the student model (S) by mimicking the class probability from a pre-trained teacher model (T) without training data. Under such setting, an ideal scenario is that T can help generate "good" samples from a generator (G) to maximally benefit S. However, existing arts suffer from the non-ideal generated samples under the disturbance of the gap (i.e., either too large or small) between the class probabilities of T and S; for example, the generated samples with too large gap may exhibit excessive information for S, while too small gap leads to the limited knowledge in the samples, resulting into the poor generalization. Meanwhile, they fail to judge the "goodness" of the generated samples for S since the fixed T is not necessarily ideal. In this paper, we aim to answer what is inside the gap box; together with how to yield "good" generated samples for DFKD? To this end, we propose a Gap-Sensitive Sample Generation (GapSSG) approach, by revisiting the empirical distilled risk from a data-free perspective, which confirms the existence of an ideal teacher (T *), while theoretically implying: (1) the gap disturbance originates from the mismatch between T and T *, hence the class probabilities of T enable the approximation to those of T *; and (2) "good" samples should maximally benefit S via T's class probabilities, owing to unknown T *. To this end, we unpack the gap box between T and S as two findings: inherent gap to perceive T and T *; derived gap to monitor S and T *. Benefiting from the derived gap that focuses on the adaptability of generated sample to S, we attempt to track student's training route (a series of training epochs) to capture the category distribution of S; upon which, a regulatory factor is further devised to approximate T * over inherent gap, so as to generate "good" samples to S. Furthermore, during the distillation process, a sample-balanced strategy comes up to tackle the overfitting and missing knowledge issues between the generated partial and critical samples by training G. The theoretical and empirical studies verify the advantages of GapSSG over the state-of-the-arts.

MVEB: Self-Supervised Learning With Multi-View Entropy Bottleneck

Self-supervised learning aims to learn representation that can be effectively generalized to downstream tasks. Many self-supervised approaches regard two views of an image as both the input and the self-supervised signals, assuming that either view contains the same task-relevant information and the shared information is (approximately) sufficient for predicting downstream tasks. Recent studies show that discarding superfluous information not shared between the views can improve generalization. Hence, the ideal representation is sufficient for downstream tasks and contains minimal superfluous information, termed minimal sufficient representation. One can learn this representation by maximizing the mutual information between the representation and the supervised view while eliminating superfluous information. Nevertheless, the computation of mutual information is notoriously intractable. In this work, we propose an objective termed multi-view entropy bottleneck (MVEB) to learn minimal sufficient representation effectively. MVEB simplifies the minimal sufficient learning to maximizing both the agreement between the embeddings of two views and the differential entropy of the embedding distribution. Our experiments confirm that MVEB significantly improves performance. For example, it achieves top-1 accuracy of 76.9% on ImageNet with a vanilla ResNet-50 backbone on linear evaluation. To the best of our knowledge, this is the new state-of-the-art result with ResNet-50.

EvHandPose: Event-Based 3D Hand Pose Estimation With Sparse Supervision

Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way. However, due to the asynchronous differential imaging mechanism, it is challenging to design event representation to encode hand motion information especially when the hands are not moving (causing motion ambiguity), and it is infeasible to fully annotate the temporally dense event stream. In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation and alleviating the motion ambiguity issue. To solve the problem under sparse annotation, we design contrast maximization and hand-edge constraints in Pose-to-IWE (Image with Warped Events) module and formulate EvHandPose in a weakly-supervision framework. We further build EvRealHands, the first large-scale real-world event-based hand pose dataset on several challenging scenes to bridge the real-synthetic domain gap. Experiments on EvRealHands demonstrate that EvHandPose outperforms previous event-based methods under all evaluation scenes, achieves accurate and stable hand pose estimation with high temporal resolution in fast motion and strong light scenes compared with RGB-based methods, generalizes well to outdoor scenes and another type of event camera, and shows the potential for the hand gesture recognition task.

Do we really need a large number of visual prompts?

Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts to input space, shows competitive fine-tuning performance compared to training of full network parameters. However, VPT increases the number of input tokens, resulting in additional computational overhead. In this paper, we analyze the impact of the number of prompts on fine-tuning performance and self-attention operation in a vision transformer architecture. Through theoretical and empirical analysis we show that adding more prompts does not lead to linear performance improvement. Further, we propose a Prompt Condensation (PC) technique that aims to prevent performance degradation from using a small number of prompts. We validate our methods on FGVC and VTAB-1k tasks and show that our approach reduces the number of prompts by ∼70% while maintaining accuracy.

Monocular BEV Perception of Road Scenes via Front-to-Top View Projection

HD map reconstruction is crucial for autonomous driving. LiDAR-based methods are limited due to expensive sensors and time-consuming computation. Camera-based methods usually need to perform road segmentation and view transformation separately, which often causes distortion and missing content. To push the limits of the technology, we present a novel framework that reconstructs a local map formed by road layout and vehicle occupancy in the bird's-eye view given a front-view monocular image only. We propose a front-to-top view projection (FTVP) module, which takes the constraint of cycle consistency between views into account and makes full use of their correlation to strengthen the view transformation and scene understanding. In addition, we apply multi-scale FTVP modules to propagate the rich spatial information of low-level features to mitigate spatial deviation of the predicted object location. Experiments on public benchmarks show that our method achieves various tasks on road layout estimation, vehicle occupancy estimation, and multi-class semantic estimation, at a performance level comparable to the state-of-the-arts, while maintaining superior efficiency.

Generative Variational-Contrastive Learning for Self-Supervised Point Cloud Representation

Self-supervised representation learning for 3D point clouds has attracted increasing attention. However, existing methods in the field of 3D computer vision generally use fixed embeddings to represent the latent features, and impose hard constraints on the embeddings to make the latent feature values of the positive samples converge to consistency, which limits the ability of feature extractors to generalize over different data domains. To address this issue, we propose a Generative Variational-Contrastive Learning (GVC) model, where Gaussian distribution is used to construct a continuous, smoothed representation of the latent features. A distribution constraint and cross-supervision are constructed to improve the transfer ability of the feature extractor over synthetic and real-world data. Specifically, we design a variational contrastive module to constrain the feature distribution instead of feature values corresponding to each sample in the latent space. Moreover, a generative cross-supervision module is introduced to preserve the invariance features and promote the consistency of feature distribution among positive samples. Experimental results demonstrate that GVC achieves SOTA on different downstream tasks. In particular, with only pre-training on the synthetic dataset, GVC achieves a lead of 8.4% and 14.2% when transferring to the real-world dataset in the linear classification and few-shot classification.

Elevation Resolution Enhancement Oriented 3D Ultrasound Imaging

Three-dimensional (3D) ultrasound imaging can be accomplished by reconstructing a sequence of two-dimensional (2D) ultrasound images. However, 2D ultrasound images usually suffer from low resolution in the elevation direction, thereby impacting the accuracy of 3D reconstructed results. The lateral resolution of 2D ultrasound is known to significantly exceed the elevation resolution. By combining scanning sequences acquired from orthogonal directions, the effects of poor elevation resolution can be mitigated through a composite reconstructing process. Moreover, capturing ultrasound images from multiple perspectives necessitates a precise probe positioning method with a wide angle of coverage. Optical tracking is popularly used for probe positioning for its high accuracy and environment-robustness. In this paper, a novel large-angle accurate optical positioning method is used for enhancing resolution in 3D ultrasound imaging through orthogonal-view scanning and composite reconstruction. Experiments on two phantoms proved that our method could significantly improve reconstruction accuracy in the elevation direction of the probe compared with single-angle parallel scanning. The results indicate that our method holds the potential to improve current 3D ultrasound imaging techniques.

Applications of machine learning in urodynamics: A narrative review

BACKGROUND

Machine learning algorithms as a research tool, including traditional machine learning and deep learning, are increasingly applied to the field of urodynamics. However, no studies have evaluated how to select appropriate algorithm models for different urodynamic research tasks.

METHODS

We undertook a narrative review evaluating how the published literature reports the applications of machine learning in urodynamics. We searched PubMed up to December 2023, limited to the English language. We selected the following search terms: artificial intelligence, machine learning, deep learning, urodynamics, and lower urinary tract symptoms. We identified three domains for assessment in advance of commencing the review. These were the applications of urodynamic studies examination, applications of diagnoses of dysfunction related to urodynamics, and applications of prognosis prediction.

RESULTS

The machine learning algorithm applied in the field of urodynamics can be mainly divided into three aspects, which are urodynamic examination, diagnosis of urinary tract dysfunction and prediction of the efficacy of various treatment methods. Most of these studies were single-center retrospective studies, lacking external validation, requiring further validation of model generalization ability, and insufficient sample size. The relevant research in this field is still in the preliminary exploration stage; there are few high-quality multi-center clinical studies, and the performance of various models still needs to be further optimized, and there is still a distance from clinical application.

CONCLUSIONS

At present, there is no research to summarize and analyze the machine learning algorithms applied in the field of urodynamics. The purpose of this review is to summarize and classify the machine learning algorithms applied in this field and to guide researchers to select the appropriate algorithm model for different task requirements to achieve the best results.

Recognition of diabetic retinopathy and macular edema using deep learning

Diabetic retinopathy (DR) and diabetic macular edema (DME) are both serious eye conditions associated with diabetes and if left untreated, and they can lead to permanent blindness. Traditional methods for screening these conditions rely on manual image analysis by experts, which can be time-consuming and costly due to the scarcity of such experts. To overcome the aforementioned challenges, we present the Modified CornerNet approach with DenseNet-100. This system aims to localize and classify lesions associated with DR and DME. To train our model, we first generate annotations for input samples. These annotations likely include information about the location and type of lesions within the retinal images. DenseNet-100 is a deep CNN used for feature extraction, and CornerNet is a one-stage object detection model. CornerNet is known for its ability to accurately localize small objects, which makes it suitable for detecting lesions in retinal images. We assessed our technique on two challenging datasets, EyePACS and IDRiD. These datasets contain a diverse range of retinal images, which is important to estimate the performance of our model. Further, the proposed model is also tested in the cross-corpus scenario on two challenging datasets named APTOS-2019 and Diaretdb1 to assess the generalizability of our system. According to the accomplished analysis, our method outperformed the latest approaches in terms of both qualitative and quantitative results. The ability to effectively localize small abnormalities and handle over-fitted challenges is highlighted as a key strength of the suggested framework which can assist the practitioners in the timely recognition of such eye ailments.

Intersection-union dual-stream cross-attention Lova-SwinUnet for skin cancer hair segmentation and image repair

Skin cancer images have hair occlusion problems, which greatly affects the accuracy of diagnosis and classification. Current dermoscopic hair removal methods use segmentation networks to locate hairs, and then uses repair networks to perform image repair. However, it is difficult to segment hair and capture the overall structure between hairs because of the hair being thin, unclear, and similar in color to the entire image. When conducting image restoration tasks, the only available images are those obstructed by hair, and there is no corresponding ground truth (supervised data) of the same scene without hair obstruction. In addition, the texture information and structural information used in existing repair methods are often insufficient, which leads to poor results in skin cancer image repair. To address these challenges, we propose the intersection-union dual-stream cross-attention Lova-SwinUnet (IUDC-LS). Firstly, we propose the Lova-SwinUnet module, which embeds Lovasz loss function into Swin-Unet, enabling the network to better capture features of various scales, thus obtaining better hair mask segmentation results. Secondly, we design the intersection-union (IU) module, which takes the mask results obtained in the previous step for pairwise intersection or union, and then overlays the results on the skin cancer image without hair to generate the labeled training data. This turns the unsupervised image repair task into the supervised one. Finally, we propose the dual-stream cross-attention (DC) module, which makes texture information and structure information interact with each other, and then uses cross-attention to make the network pay attention to the more important texture information and structure information in the fusion process of texture information and structure information, so as to improve the effect of image repair. The experimental results show that the PSNR index and SSIM index of the proposed method are increased by 5.4875 and 0.0401 compared with the other common methods. Experimental results unequivocally demonstrate the effectiveness of our approach, which serves as a potent tool for skin cancer detection, significantly surpassing the performance of comparable methods.

HyperSOR: Context-Aware Graph Hypernetwork for Salient Object Ranking

Salient object ranking (SOR) aims to segment salient objects in an image and simultaneously predict their saliency rankings, according to the shifted human attention over different objects. The existing SOR approaches mainly focus on object-based attention, e.g., the semantic and appearance of object. However, we find that the scene context plays a vital role in SOR, in which the saliency ranking of the same object varies a lot at different scenes. In this paper, we thus make the first attempt towards explicitly learning scene context for SOR. Specifically, we establish a large-scale SOR dataset of 24,373 images with rich context annotations, i.e., scene graphs, segmentation, and saliency rankings. Inspired by the data analysis on our dataset, we propose a novel graph hypernetwork, named HyperSOR, for context-aware SOR. In HyperSOR, an initial graph module is developed to segment objects and construct an initial graph by considering both geometry and semantic information. Then, a scene graph generation module with multi-path graph attention mechanism is designed to learn semantic relationships among objects based on the initial graph. Finally, a saliency ranking prediction module dynamically adopts the learned scene context through a novel graph hypernetwork, for inferring the saliency rankings. Experimental results show that our HyperSOR can significantly improve the performance of SOR.

Scalable Video Object Segmentation With Identification Mechanism

This paper delves into the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object Segmentation (VOS). Previous VOS methods decode features with a single positive object, limiting the learning of multi-object representation as they must match and segment each target separately under multi-object scenarios. Additionally, earlier techniques catered to specific application objectives and lacked the flexibility to fulfill different speed-accuracy requirements. To address these problems, we present two innovative approaches, Associating Objects with Transformers (AOT) and Associating Objects with Scalable Transformers (AOST). In pursuing effective multi-object modeling, AOT introduces the IDentification (ID) mechanism to allocate each object a unique identity. This approach enables the network to model the associations among all objects simultaneously, thus facilitating the tracking and segmentation of objects in a single network pass. To address the challenge of inflexible deployment, AOST further integrates scalable long short-term transformers that incorporate scalable supervision and layer-wise ID-based attention. This enables online architecture scalability in VOS for the first time and overcomes ID embeddings' representation limitations. Given the absence of a benchmark for VOS involving densely multi-object annotations, we propose a challenging Video Object Segmentation in the Wild (VOSW) benchmark to validate our approaches. We evaluated various AOT and AOST variants using extensive experiments across VOSW and five commonly used VOS benchmarks, including YouTube-VOS 2018 & 2019 Val, DAVIS-2017 Val & Test, and DAVIS-2016. Our approaches surpass the state-of-the-art competitors and display exceptional efficiency and scalability consistently across all six benchmarks. Moreover, we notably achieved the 1st position in the 3 rd Large-scale Video Object Segmentation Challenge.

MCF-SMSIS: Multi-tasking with complementary functions for stereo matching and surgical instrument segmentation

Stereo matching and instrument segmentation of laparoscopic surgical scenarios are key tasks in robotic surgical automation. Many researchers have been studying the two tasks separately for stereo matching and instrument segmentation. However, the relationship between these two tasks is often neglected. In this paper, we propose a model framework for multi-tasking with complementary functions for stereo matching and surgical instrument segmentation (MCF-SMSIS). We aim to complement the features of instrument prediction segmentation to the parallax matching block of stereo matching. We also propose two new evaluation metrics (MINPD and MAXPD) for assessing how well the parallax range matches the migrated domain when the model used for the stereo matching task undergoes domain migration. We performed stereo matching experiments on the SCARED , SERV-CT dataset as well as instrumentation segmentation experiments on the AutoLaparo dataset. The results demonstrate the effectiveness of the proposed method. In particular, stereo matching supplemented with instrument features reduced EPE, >3px and RMSE Depth in the surgical instrument section by 9.5%, 12.7% and 6.51%, respectively. The instrumentation segmentation performance also achieves a DSC value of 0.9233. Moreover, MCF-SMSIS takes only 0.14 s to infer a set of images. The model code and model weights for each stage are available from https://github.com/wurenkai/MCF-SMSIS.

Gradient-Based Instance-Specific Visual Explanations for Object Specification and Object Discrimination

We propose the gradient-weighted Object Detector Activation Maps (ODAM), a visual explanation technique for interpreting the predictions of object detectors. Utilizing the gradients of detector targets flowing into the intermediate feature maps, ODAM produces heat maps that show the influence of regions on the detector's decision for each predicted attribute. Compared to previous works on classification activation maps (CAM), ODAM generates instance-specific explanations rather than class-specific ones. We show that ODAM is applicable to one-stage, two-stage, and transformer-based detectors with different types of detector backbones and heads, and produces higher-quality visual explanations than the state-of-the-art in terms of both effectiveness and efficiency. We discuss two explanation tasks for object detection: 1) object specification: what is the important region for the prediction? 2) object discrimination: which object is detected? Aiming at these two aspects, we present a detailed analysis of the visual explanations of detectors and carry out extensive experiments to validate the effectiveness of the proposed ODAM. Furthermore, we investigate user trust on the explanation maps, how well the visual explanations of object detectors agrees with human explanations, as measured through human eye gaze, and whether this agreement is related with user trust. Finally, we also propose two applications, ODAM-KD and ODAM-NMS, based on these two abilities of ODAM. ODAM-KD utilizes the object specification of ODAM to generate top-down attention for key predictions and instruct the knowledge distillation of object detection. ODAM-NMS considers the location of the model's explanation for each prediction to distinguish the duplicate detected objects. A training scheme, ODAM-Train, is proposed to improve the quality on object discrimination, and help with ODAM-NMS.

Unsupervised domain adaptive segmentation algorithm based on two-level category alignment

To enhance the model's generalization ability in unsupervised domain adaptive segmentation tasks, most approaches have primarily focused on pixel-level local features, but neglected the clue in category information. This limitation results in the segmentation network only learning global inter-domain invariant features but ignoring the category-specific inter-domain invariant features, which degenerates the segmentation performance. To address this issue, we present an Unsupervised Domain Adaptive algorithm based on two-level Category Alignment in two different spaces for semantic segmentation tasks, denoted as UDAca+. The first level is image-level category alignment based on class activation map (CAM), and the second one is pixel-level category alignment based on pseudo label. By utilizing category information, UDAca+ can effectively capture domain-invariant yet category-discriminative feature representations to improve segmentation accuracy. In addition, an adversarial learning-based strategy in mixed domain is designed to train the proposed network. Moreover, a confidence calculation method is introduced to mitigate the misleading issues of negative transfer and over-alignment caused by the noise in image-level pseudo labels. UDAca+ achieves the state-of-the-art (SOTA) performance on two synthetic-to-real adaptative tasks, and verifies its effectiveness for image segmentation.

Lightness constancy in reality, in virtual reality, and on flat-panel displays

Virtual reality (VR) displays are being used in an increasingly wide range of applications. However, previous work shows that viewers often perceive scene properties very differently in real and virtual environments and so realistic perception of virtual stimuli should always be a carefully tested conclusion, not an assumption. One important property for realistic scene perception is surface color. To evaluate how well virtual platforms support realistic perception of achromatic surface color, we assessed lightness constancy in a physical apparatus with real lights and surfaces, in a commercial VR headset, and on a traditional flat-panel display. We found that lightness constancy was good in all three environments, though significantly better in the real environment than on the flat-panel display. We also found that variability across observers was significantly greater in VR and on the flat-panel display than in the physical environment. We conclude that these discrepancies should be taken into account in applications where realistic perception is critical but also that in many cases VR can be used as a flexible alternative to flat-panel displays and a reasonable proxy for real environments.

Deep Learning for Visual Speech Analysis: A Survey

Visual speech, referring to the visual domain of speech, has attracted increasing attention due to its wide applications, such as public security, medical treatment, military defense, and film entertainment. As a powerful AI strategy, deep learning techniques have extensively promoted the development of visual speech learning. Over the past five years, numerous deep learning based methods have been proposed to address various problems in this area, especially automatic visual speech recognition and generation. To push forward future research on visual speech, this paper will present a comprehensive review of recent progress in deep learning methods on visual speech analysis. We cover different aspects of visual speech, including fundamental problems, challenges, benchmark datasets, a taxonomy of existing methods, and state-of-the-art performance. Besides, we also identify gaps in current research and discuss inspiring future research directions.

Next-Generation Tear Meniscus Height Detecting and Measuring Smartphone-Based Deep Learning Algorithm Leads in Dry Eye Management

Purpose

This study aims to develop and assess an infrastructure using Python-based deep learning code for future diagnostic and management purposes related to dry eye disease (DED) utilizing smartphone images.

Design

Cross-sectional study using data which was gathered in Vision Health Research Clinic.

Participants

One thousand twenty-one eye images from 734 patients were included in this article that categorizes into 70% females and 30% males, with no sex and age limit.

Methods

One specialist captured eye images using Samsung A71 (601 images) and iPhone 11 (420 images) cell phones with the flashlight on and direct gaze to the camera. These images include the area of only 1 eye (left/right).

Main Outcome Measures

First, our specialist did 3 different segmentations for every eye image separately for 80% of the training data. This part contains eye, lower eyelid, and iris segmentation. In 20% of test data after automated cropping of the lower eyelid margin and upscaling by 8×, the appropriate tear meniscus height segmentation will be chosen and measured using a deep learning algorithm.

Results

The model was trained on 80% of the data and 20% of the data used for validation from both phones with different resolutions. The dice coefficient of the trained model for validation data is 98.68%, and the accuracy of the overall model is 95.39%.

Conclusions

It appears that this algorithm holds the potential to herald an evolution in the future of diagnosis and management of DED by homecare devices solely through smartphones.

Financial Disclosures

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Turning a CLIP Model Into a Scene Text Spotter

We exploit the potential of the large-scale Contrastive Language-Image Pretraining (CLIP) model to enhance scene text detection and spotting tasks, transforming it into a robust backbone, FastTCM-CR50. This backbone utilizes visual prompt learning and cross-attention in CLIP to extract image and text-based prior knowledge. Using predefined and learnable prompts, FastTCM-CR50 introduces an instance-language matching process to enhance the synergy between image and text embeddings, thereby refining text regions. Our Bimodal Similarity Matching (BSM) module facilitates dynamic language prompt generation, enabling offline computations and improving performance. FastTCM-CR50 offers several advantages: 1) It can enhance existing text detectors and spotters, improving performance by an average of 1.6% and 1.5%, respectively. 2) It outperforms the previous TCM-CR50 backbone, yielding an average improvement of 0.2% and 0.55% in text detection and spotting tasks, along with a 47.1% increase in inference speed. 3) It showcases robust few-shot training capabilities. Utilizing only 10% of the supervised data, FastTCM-CR50 improves performance by an average of 26.5% and 4.7% for text detection and spotting tasks, respectively. 4) It consistently enhances performance on out-of-distribution text detection and spotting datasets, particularly the NightTime-ArT subset from ICDAR2019-ArT and the DOTA dataset for oriented object detection.

Nondestructive detection of SSC in multiple pear (Pyrus pyrifolia Nakai) cultivars using Vis-NIR spectroscopy coupled with the Grad-CAM method

Vis-NIR spectroscopy coupled with chemometric models is frequently used for pear soluble solid content (SSC) prediction. However, the model robustness is challenged by the variations in pear cultivars. This study explored the feasibility of developing universal models for predicting SSC of multiple pear varieties to improve the model's generalizability. The mature fruits of 6 pear cultivars with green skin (Pyrus pyrifolia Nakai cv. 'Cuiyu', 'Sucui No.1' and 'Cuiguan') and brown skin (Pyrus pyrifolia Nakai cv. 'Hosui','Syusui' and 'Wakahikari') were used to establish single-cultivar models and multi-cultivar universal models using convolutional neural network (CNN), partial least square (PLS), and support vector regression (SVR) approaches. Multi-cultivar universal models were built using full spectra and important variables extracted by gradient-weighted class activation mapping (Grad-CAM), respectively. The universal models based on important variables obtained satisfactory performances with RMSEPs of 0.76, 0.59, 0.80, 1.64, 0.98, and 1.03°Brix on 6 cultivars, respectively.

Modeling multiple-criterion diagnoses by heterogeneous-instance logistic regression

Mild cognitive impairment (MCI) is a prodromal stage of Alzheimer's disease (AD) that causes a significant burden in caregiving and medical costs. Clinically, the diagnosis of MCI is determined by the impairment statuses of five cognitive domains. If one of these cognitive domains is impaired, the patient is diagnosed with MCI, and if two out of the five domains are impaired, the patient is diagnosed with AD. In medical records, most of the time, the diagnosis of MCI/AD is given, but not the statuses of the five domains. We may treat the domain statuses as missing variables. This diagnostic procedure relates MCI/AD status modeling to multiple-instance learning, where each domain resembles an instance. However, traditional multiple-instance learning assumes common predictors among instances, but in our case, each domain is associated with different predictors. In this article, we generalized the multiple-instance logistic regression to accommodate the heterogeneity in predictors among different instances. The proposed model is dubbed heterogeneous-instance logistic regression and is estimated via the expectation-maximization algorithm because of the presence of the missing variables. We also derived two variants of the proposed model for the MCI and AD diagnoses. The proposed model is validated in terms of its estimation accuracy, latent status prediction, and robustness via extensive simulation studies. Finally, we analyzed the National Alzheimer's Coordinating Center-Uniform Data Set using the proposed model and demonstrated its potential.

Developmentally stable representations of naturalistic image structure in macaque visual cortex

To determine whether post-natal improvements in form vision result from changes in mid-level visual cortex, we studied neuronal and behavioral responses to texture stimuli that were matched in local spectral content but varied in "naturalistic" structure. We made longitudinal measurements of visual behavior from 16 to 95 weeks of age, and of neural responses from 20 to 56 weeks. We also measured behavioral and neural responses in near-adult animals more than 3 years old. Behavioral sensitivity reached half-maximum around 25 weeks of age, but neural sensitivities remained stable through all ages tested. Neural sensitivity to naturalistic structure was highest in V4, lower in V2 and inferotemporal cortex (IT), and barely discernible in V1. Our results show a dissociation between stable neural performance and improving behavioral performance, which may reflect improved processing capacity in circuits downstream of visual cortex.

Classification of optic neuritis in neuromyelitis optica spectrum disorders (NMOSD) on MRI using CNN with transfer learning and manipulation of pre-processing on augmentation

Neuromyelitis optica spectrum disorder (NMOSD), also known as Devic disease, is an autoimmune central nervous system disorder in humans that commonly causes inflammatory demyelination in the optic nerves and spinal cord. Inflammation in the optic nerves is termed optic neuritis (ON). ON is a common clinical presentation; however, it is not necessarily present in all NMOSD patients. ON in NMOSD can be relapsing and result in severe vision loss. To the best of our knowledge, no study utilises deep learning to classify ON changes on MRI among patients with NMOSD. Therefore, this study aims to deploy eight state-of-the-art CNN models (Inception-v3, Inception-ResNet-v2, ResNet-101, Xception, ShuffleNet, DenseNet-201, MobileNet-v2, and EfficientNet-B0) with transfer learning to classify NMOSD patients with and without chronic ON using optic nerve magnetic resonance imaging. This study also investigated the effects of data augmentation before and after dataset splitting on cropped and whole images. Both quantitative and qualitative assessments (with Grad-Cam) were used to evaluate the performances of the CNN models. The Inception-v3 was identified as the best CNN model for classifying ON among NMOSD patients, with accuracy of 99.5%, sensitivity of 98.9%, specificity of 93.0%, precision of 100%, NPV of 99.0%, and F1-score of 99.4%. This study also demonstrated that the application of augmentation after dataset splitting could avoid information leaking into the testing datasets, hence producing more realistic and reliable results.

Computing the alpha complex using dual active set quadratic programming

The alpha complex is a fundamental data structure from computational geometry, which encodes the topological type of a union of balls B ( x ; r ) ⊂ R m for x ∈ S , including a weighted version that allows for varying radii. It consists of the collection of "simplices" σ = { x 0 , . . . , x k } ⊂ S , which correspond to nonempty ( k + 1 ) -fold intersections of cells in a radius-restricted version of the Voronoi diagramVor ( S , r ) . Existing algorithms for computing the alpha complex require that the points reside in low dimension because they begin by computing the entire Delaunay complex, which rapidly becomes intractable, even when the alpha complex is of a reasonable size. This paper presents a method for computing the alpha complex without computing the full Delaunay triangulation by applying Lagrangian duality, specifically an algorithm based on dual quadratic programming that seeks to rule simplices out rather than ruling them in.

Self-supervised context-aware correlation filter for robust landmark tracking in liver ultrasound sequences

OBJECTIVES

Respiratory motion-induced displacement of internal organs poses a significant challenge in image-guided radiation therapy, particularly affecting liver landmark tracking accuracy.

METHODS

Addressing this concern, we propose a self-supervised method for robust landmark tracking in long liver ultrasound sequences. Our approach leverages a Siamese-based context-aware correlation filter network, trained by using the consistency loss between forward tracking and back verification. By effectively utilizing both labeled and unlabeled liver ultrasound images, our model, Siam-CCF , mitigates the impact of speckle noise and artifacts on ultrasonic image tracking by a context-aware correlation filter. Additionally, a fusion strategy for template patch feature helps the tracker to obtain rich appearance information around the point-landmark.

RESULTS

Siam-CCF achieves a mean tracking error of 0.79 ± 0.83 mm at a frame rate of 118.6 fps, exhibiting a superior speed-accuracy trade-off on the public MICCAI 2015 Challenge on Liver Ultrasound Tracking (CLUST2015) 2D dataset. This performance won the 5th place on the CLUST2015 2D point-landmark tracking task.

CONCLUSIONS

Extensive experiments validate the effectiveness of our proposed approach, establishing it as one of the top-performing techniques on the CLUST2015 online leaderboard at the time of this submission.

Microbial artists: the role of parasite microbiomes in explaining colour polymorphism among amphipods and potential link to host manipulation

Parasite infections are increasingly reported to change the microbiome of the parasitized hosts, while parasites bring their own microbes to what can be a multi-dimensional interaction. For instance, a recent hypothesis suggests that the microbial communities harboured by parasites may play a role in the well-documented ability of many parasites to manipulate host phenotype, and explain why the degree to which host phenotype is altered varies among conspecific parasites. Here, we explored whether the microbiomes of both hosts and parasites are associated with variation in host manipulation by parasites. Using colour quantification methods applied to digital images, we investigated colour variation among uninfected Transorchestia serrulata amphipods, as well as amphipods infected with Plagiorhynchus allisonae acanthocephalans and with a dilepidid cestode. We then characterized the bacteriota of amphipod hosts and of their parasites, looking for correlations between host phenotype and the bacterial taxa associated with hosts and parasites. We found large variation in amphipod colours, and weak support for a direct impact of parasites on the colour of their hosts. Conversely, and most interestingly, the parasite's bacteriota was more strongly correlated with colour variation among their amphipod hosts, with potential impact of amphipod-associated bacteria as well. Some bacterial taxa found associated with amphipods and parasites may have the ability to synthesize pigments, and we propose they may interact with colour determination in the amphipods. This study provides correlational support for an association between the parasite's microbiome and the evolution of host manipulation by parasites and host-parasite interactions more generally.

Deep learning based approach: automated gingival inflammation grading model using gingival removal strategy

Gingival inflammation grade serves as a well-established index in periodontitis. The aim of this study was to develop a deep learning network utilizing a novel feature extraction method for the automatic assessment of gingival inflammation. T-distributed Stochastic Neighbor Embedding (t-SNE) was utilized for dimensionality reduction. A convolutional neural network (CNN) model based on DenseNet was developed for the identification and evaluation of gingival inflammation. To enhance the performance of the deep learning (DL) model, a novel teeth removal algorithm was implemented. Additionally, a Grad-CAM +  + encoder was applied to generate heatmaps for computer visual attention analysis. The mean Intersection over Union (MIoU) for the identification of gingivitis was 0.727 ± 0.117. The accuracy rates for the five inflammatory degrees were 77.09%, 77.25%, 74.38%, 73.68% and 79.22%. The Area Under the Receiver Operating Characteristic (AUROC) values were 0.83, 0.80, 0.81, 0.81 and 0.84, respectively. The attention ratio towards gingival tissue increased from 37.73% to 62.20%, and within 8 mm of the gingival margin, it rose from 21.11% to 38.23%. On the gingiva, the overall attention ratio increased from 51.82% to 78.21%. The proposed DL model with novel feature extraction method provides high accuracy and sensitivity for identifying and grading gingival inflammation.

Recognition of Forward Head Posture Through 3D Human Pose Estimation With a Graph Convolutional Network: Development and Feasibility Study

BACKGROUND

Prolonged improper posture can lead to forward head posture (FHP), causing headaches, impaired respiratory function, and fatigue. This is especially relevant in sedentary scenarios, where individuals often maintain static postures for extended periods-a significant part of daily life for many. The development of a system capable of detecting FHP is crucial, as it would not only alert users to correct their posture but also serve the broader goal of contributing to public health by preventing the progression of chronic injuries associated with this condition. However, despite significant advancements in estimating human poses from standard 2D images, most computational pose models do not include measurements of the craniovertebral angle, which involves the C7 vertebra, crucial for diagnosing FHP.

OBJECTIVE

Accurate diagnosis of FHP typically requires dedicated devices, such as clinical postural assessments or specialized imaging equipment, but their use is impractical for continuous, real-time monitoring in everyday settings. Therefore, developing an accessible, efficient method for regular posture assessment that can be easily integrated into daily activities, providing real-time feedback, and promoting corrective action, is necessary.

METHODS

The system sequentially estimates 2D and 3D human anatomical key points from a provided 2D image, using the Detectron2D and VideoPose3D algorithms, respectively. It then uses a graph convolutional network (GCN), explicitly crafted to analyze the spatial configuration and alignment of the upper body's anatomical key points in 3D space. This GCN aims to implicitly learn the intricate relationship between the estimated 3D key points and the correct posture, specifically to identify FHP.

RESULTS

The test accuracy was 78.27% when inputs included all joints corresponding to the upper body key points. The GCN model demonstrated slightly superior balanced performance across classes with an F1-score (macro) of 77.54%, compared to the baseline feedforward neural network (FFNN) model's 75.88%. Specifically, the GCN model showed a more balanced precision and recall between the classes, suggesting its potential for better generalization in FHP detection across diverse postures. Meanwhile, the baseline FFNN model demonstrates a higher precision for FHP cases but at the cost of lower recall, indicating that while it is more accurate in confirming FHP when detected, it misses a significant number of actual FHP instances. This assertion is further substantiated by the examination of the latent feature space using t-distributed stochastic neighbor embedding, where the GCN model presented an isotropic distribution, unlike the FFNN model, which showed an anisotropic distribution.

CONCLUSIONS

Based on 2D image input using 3D human pose estimation joint inputs, it was found that it is possible to learn FHP-related features using the proposed GCN-based network to develop a posture correction system. We conclude the paper by addressing the limitations of our current system and proposing potential avenues for future work in this area.

Estimating biological age from retinal imaging: a scoping review

BACKGROUND/AIMS

The emerging concept of retinal age, a biomarker derived from retinal images, holds promise in estimating biological age. The retinal age gap (RAG) represents the difference between retinal age and chronological age, which serves as an indicator of deviations from normal ageing. This scoping review aims to collate studies on retinal age to determine its potential clinical utility and to identify knowledge gaps for future research.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist, eligible non-review, human studies were identified, selected and appraised. PubMed, Scopus, SciELO, PsycINFO, Google Scholar, Cochrane, CINAHL, Africa Wide EBSCO, MedRxiv and BioRxiv databases were searched to identify literature pertaining to retinal age, the RAG and their associations. No restrictions were imposed on publication date.

RESULTS

Thirteen articles published between 2022 and 2023 were analysed, revealing four models capable of determining biological age from retinal images. Three models, 'Retinal Age', 'EyeAge' and a 'convolutional network-based model', achieved comparable mean absolute errors: 3.55, 3.30 and 3.97, respectively. A fourth model, 'RetiAGE', predicting the probability of being older than 65 years, also demonstrated strong predictive ability with respect to clinical outcomes. In the models identified, a higher predicted RAG demonstrated an association with negative occurrences, notably mortality and cardiovascular health outcomes.

CONCLUSION

This review highlights the potential clinical application of retinal age and RAG, emphasising the need for further research to establish their generalisability for clinical use, particularly in neuropsychiatry. The identified models showcase promising accuracy in estimating biological age, suggesting its viability for evaluating health status.

A Dataset for Detection and Segmentation of Underwater Marine Debris in Shallow Waters

Robust object detection is crucial for automating underwater marine debris collection. While supervised deep learning achieves state-of-the-art performance in discriminative tasks, replicating this success on underwater data is challenging. The generalization of these methods suffers due to a lack of available annotated data considering different sources of variation in the unstructured underwater environment and imaging conditions. In this paper, we present the Seaclear Marine Debris Dataset, the first publicly available shallow-water marine debris dataset annotated for instance segmentation/object detection. The dataset contains 8610 images collected using ROVs at multiple locations and with different cameras, annotated for 40 object categories, encompassing not only litter but also observed animals, plants, and robot parts. As part of the technical validation, we provide baseline results for object detection using Faster RCNN and YOLOv6 models. Furthermore, we demonstrate the non-triviality of generalizing the trained model performance to unseen sites and cameras due to domain shift. This underscores the value of the presented dataset in further developing robust models for underwater debris detection.

Enhancing YOLO for occluded vehicle detection with grouped orthogonal attention and dense object repulsion

In real-life complex traffic environments, vehicles are often occluded by extraneous background objects and other vehicles, leading to severe degradation of object detector performance. To address this issue, we propose a method named YOLO-OVD (YOLO for occluded vehicle detection) and a dataset for effectively handling vehicle occlusion in various scenarios. To highlight the model attention in unobstructed region of vehicles, we design a novel grouped orthogonal attention (GOA) module to achieve maximum information extraction between channels. We utilize grouping and channel shuffling to address the initialization and computational issues of original orthogonal filters, followed by spatial attention for enhancing spatial features in vehicle-visible regions. We introduce a CIoU-based repulsion term into the loss function to augment the network's localization accuracy in scenarios involving densely packed vehicles. Moreover, we explore the effect of the knowledge-based Laplacian Pyramid on the OVD performance, which contributes to fast convergence in training and ensures more detailed and comprehensive feature retention. We conduct extensive experiments on the established occluded vehicle detection dataset, which demonstrates that the proposed YOLO-OVD model significantly outperforms 14 representative object detectors. Notably, it achieves improvements of 4.7% in Precision, 3.6% in AP@0.5, and 1.9% in AP@0.5:0.95 compared to the YOLOv5 baseline.

Leaf rolling detection in maize under complex environments using an improved deep learning method

Leaf rolling is a common adaptive response that plants have evolved to counteract the detrimental effects of various environmental stresses. Gaining insight into the mechanisms underlying leaf rolling alterations presents researchers with a unique opportunity to enhance stress tolerance in crops exhibiting leaf rolling, such as maize. In order to achieve a more profound understanding of leaf rolling, it is imperative to ascertain the occurrence and extent of this phenotype. While traditional manual leaf rolling detection is slow and laborious, research into high-throughput methods for detecting leaf rolling within our investigation scope remains limited. In this study, we present an approach for detecting leaf rolling in maize using the YOLOv8 model. Our method, LRD-YOLO, integrates two significant improvements: a Convolutional Block Attention Module to augment feature extraction capabilities, and a Deformable ConvNets v2 to enhance adaptability to changes in target shape and scale. Through experiments on a dataset encompassing severe occlusion, variations in leaf scale and shape, and complex background scenarios, our approach achieves an impressive mean average precision of 81.6%, surpassing current state-of-the-art methods. Furthermore, the LRD-YOLO model demands only 8.0 G floating point operations and the parameters of 3.48 M. We have proposed an innovative method for leaf rolling detection in maize, and experimental outcomes showcase the efficacy of LRD-YOLO in precisely detecting leaf rolling in complex scenarios while maintaining real-time inference speed.

3DFFL: privacy-preserving Federated Few-Shot Learning for 3D point clouds in autonomous vehicles

This paper presents a comprehensive study of 3D point cloud Federated Few-Shot Learning (3DFFL), focusing on addressing challenges such as limited data availability and privacy concerns in point cloud classification for applications such as autonomous vehicles. We introduce a novel approach that integrates Federated Learning with Few-Shot Learning techniques, with a special emphasis on optimizing network architectures for 3D point cloud data. Our method capitalizes on the strengths of PointNet++ for feature extraction and ProtoNet for classification, all within a federated learning framework to ensure data privacy and collaborative learning. Significantly, the approach is augmented with the use of attention and SoftMax layers, enhancing the feature extraction and classification processes. Extensive experiments on the ModelNet40, ShapeNet, and ScanOnjectNN datasets validate our method's accuracy and adaptability in handling 3D point cloud classification, especially in privacy-sensitive and collaborative scenarios. This study not only demonstrates the potential of integrating attention mechanisms and SoftMax layers in 3DFFL but also lays a robust foundation for future advancements in this evolving field, particularly in technologies dependent on 3D data processing.

Reconstruct incomplete relation for incomplete modality brain tumor segmentation

Different brain tumor magnetic resonance imaging (MRI) modalities provide diverse tumor-specific information. Previous works have enhanced brain tumor segmentation performance by integrating multiple MRI modalities. However, multi-modal MRI data are often unavailable in clinical practice. An incomplete modality leads to missing tumor-specific information, which degrades the performance of existing models. Various strategies have been proposed to transfer knowledge from a full modality network (teacher) to an incomplete modality one (student) to address this issue. However, they neglect the fact that brain tumor segmentation is a structural prediction problem that requires voxel semantic relations. In this paper, we propose a Reconstruct Incomplete Relation Network (RIRN) that transfers voxel semantic relational knowledge from the teacher to the student. Specifically, we propose two types of voxel relations to incorporate structural knowledge: Class-relative relations (CRR) and Class-agnostic relations (CAR). The CRR groups voxels into different tumor regions and constructs a relation between them. The CAR builds a global relation between all voxel features, complementing the local inter-region relation. Moreover, we use adversarial learning to align the holistic structural prediction between the teacher and the student. Extensive experimentation on both the BraTS 2018 and BraTS 2020 datasets establishes that our method outperforms all state-of-the-art approaches.

Prediction of midpalatal suture maturation stage based on transfer learning and enhanced vision transformer

BACKGROUND

Maxillary expansion is an important treatment method for maxillary transverse hypoplasia. Different methods of maxillary expansion should be carried out depending on the midpalatal suture maturation levels, and the diagnosis was validated by palatal plane cone beam computed tomography (CBCT) images by orthodontists, while such a method suffered from low efficiency and strong subjectivity. This study develops and evaluates an enhanced vision transformer (ViT) to automatically classify CBCT images of midpalatal sutures with different maturation stages.

METHODS

In recent years, the use of convolutional neural network (CNN) to classify images of midpalatal suture with different maturation stages has brought positive significance to the decision of the clinical maxillary expansion method. However, CNN cannot adequately learn the long-distance dependencies between images and features, which are also required for global recognition of midpalatal suture CBCT images. The Self-Attention of ViT has the function of capturing the relationship between long-distance pixels of the image. However, it lacks the inductive bias of CNN and needs more data training. To solve this problem, a CNN-enhanced ViT model based on transfer learning is proposed to classify midpalatal suture CBCT images. In this study, 2518 CBCT images of the palate plane are collected, and the images are divided into 1259 images as the training set, 506 images as the verification set, and 753 images as the test set. After the training set image preprocessing, the CNN-enhanced ViT model is trained and adjusted, and the generalization ability of the model is tested on the test set.

RESULTS

The classification accuracy of our proposed ViT model is 95.75%, and its Macro-averaging Area under the receiver operating characteristic Curve (AUC) and Micro-averaging AUC are 97.89% and 98.36% respectively on our data test set. The classification accuracy of the best performing CNN model EfficientnetV2_S was 93.76% on our data test set. The classification accuracy of the clinician is 89.10% on our data test set.

CONCLUSIONS

The experimental results show that this method can effectively complete CBCT images classification of midpalatal suture maturation stages, and the performance is better than a clinician. Therefore, the model can provide a valuable reference for orthodontists and assist them in making correct a diagnosis.

Single image deraining via wide rectangular regional blocks and dual attention complementary enhancement network

Rain is a common weather phenomenon, and the challenge of removing rain streaks from a single image is crucial due to its detrimental impact on image quality and the extraction of valuable background information. Existing methods commonly rely on specific assumptions regarding rain models, which restricts their ability to accommodate a wide range of real-world scenarios. To overcome this limitation, these methods often require complex optimization techniques or stepwise refinement strategies. In this paper, we propose a novel wide rectangular regional block and dual attention complementary enhancement deraining kernel prediction subnet to meet the challenge. The network called WRRDANet consists of a kernel prediction subnet and pixel-wise dilation filtering. In the kernel prediction subnet, we capture more specific contextual background information and complex pixel-wise kernels. Afterward, the learned pixel-wise multi-scale kernels from the kernel prediction subnet are used to perform dilation filtering on the original rainy image, effectively restoring richer background details by expanding the scope of deraining to a larger extent. We conducted a comprehensive evaluation using synthetic and real rainfall datasets to demonstrate the effectiveness of our approach. The results, both qualitatively and quantitatively, indicate that our approach outperforms other popular rain removal methods.

Cucumber pathogenic spores' detection using the GCS-YOLOv8 network with microscopic images in natural scenes

Fungal diseases are the main factors affecting the quality and production of vegetables. Rapid and accurate detection of pathogenic spores is of great practical significance for early prediction and prevention of diseases. However, there are some problems with microscopic images collected in the natural environment, such as complex backgrounds, more disturbing materials, small size of spores, and various forms. Therefore, this study proposed an improved detection method of GCS-YOLOv8 (Global context and CARFAE and Small detector-optimized YOLOv8), effectively improving the detection accuracy of small-target pathogen spores in natural scenes. Firstly, by adding a small target detection layer in the network, the network's sensitivity to small targets is enhanced, and the problem of low detection accuracy of the small target is effectively improved. Secondly, Global Context attention is introduced in Backbone to optimize the CSPDarknet53 to 2-Stage FPN (C2F) module and model global context information. At the same time, the feature up-sampling module Content-Aware Reassembly of Features (CARAFE) was introduced into Neck to enhance the ability of the network to extract spore features in natural scenes further. Finally, we used an Explainable Artificial Intelligence (XAI) approach to interpret the model's predictions. The experimental results showed that the improved GCS-YOLOv8 model could detect the spores of the three fungi with an accuracy of 0.926 and a model size of 22.8 MB, which was significantly superior to the existing model and showed good robustness under different brightness conditions. The test on the microscopic images of the infection structure of cucumber down mildew also proved that the model had good generalization. Therefore, this study realized the accurate detection of pathogen spores in natural scenes and provided feasible technical support for early predicting and preventing fungal diseases.

Medical image registration and its application in retinal images: a review

Medical image registration is vital for disease diagnosis and treatment with its ability to merge diverse information of images, which may be captured under different times, angles, or modalities. Although several surveys have reviewed the development of medical image registration, they have not systematically summarized the existing medical image registration methods. To this end, a comprehensive review of these methods is provided from traditional and deep-learning-based perspectives, aiming to help audiences quickly understand the development of medical image registration. In particular, we review recent advances in retinal image registration, which has not attracted much attention. In addition, current challenges in retinal image registration are discussed and insights and prospects for future research provided.