Application of computer tongue image analysis technology in the diagnosis of NAFLD

Developing tongue coating status assessment using image recognition with deep learning

PURPOSE

To build an image recognition network to evaluate tongue coating status.

METHODS

Two image recognition networks were built: one for tongue detection and another for tongue coating classification. Digital tongue photographs were used to develop both networks; images from 251 (178 women, 74.7±6.6 years) and 144 older adults (83 women, 73.8±7.3 years) who volunteered to participate were used for the tongue detection network and coating classification network, respectively. The learning objective of the tongue detection network is to extract a rectangular region that includes the tongue. You-Only-Look-Once (YOLO) v2 was used as the detection network, and transfer learning was performed using ResNet-50. The accuracy was evaluated by calculating the intersection over the union. For tongue coating classification, the rectangular area including the tongue was divided into a grid of 7×7. Five experienced panelists scored the tongue coating in each area using one of five grades, and the tongue coating index (TCI) was calculated. Transfer learning for tongue coating grades was performed using ResNet-18, and the TCI was calculated. Agreement between the panelists and network for the tongue coating grades in each area and TCI was evaluated using the kappa coefficient and intraclass correlation, respectively.

RESULTS

The tongue detection network recognized the tongue with a high intersection over union (0.885±0.081). The tongue coating classification network showed high agreement with tongue coating grades and TCI, with a kappa coefficient of 0.826 and an intraclass correlation coefficient of 0.807, respectively.

CONCLUSIONS

Image recognition enables simple and detailed assessment of tongue coating status.

S5Utis: Structured State-Space Sequence SegNeXt UNet-like Tongue Image Segmentation in Traditional Chinese Medicine

Intelligent Traditional Chinese Medicine can provide people with a convenient way to participate in daily health care. The ease of acceptance of Traditional Chinese Medicine is also a major advantage in promoting health management. In Traditional Chinese Medicine, tongue imaging is an important step in the examination process. The segmentation and processing of the tongue image directly affects the results of intelligent Traditional Chinese Medicine diagnosis. As intelligent Traditional Chinese Medicine continues to develop, remote diagnosis and patient participation will play important roles. Smartphone sensor cameras can provide irreplaceable data collection capabilities in enhancing interaction in smart Traditional Chinese Medicine. However, these factors lead to differences in the size and quality of the captured images due to factors such as differences in shooting equipment, professionalism of the photographer, and the subject's cooperation. Most current tongue image segmentation algorithms are based on data collected by professional tongue diagnosis instruments in standard environments, and are not able to demonstrate the tongue image segmentation effect in complex environments. Therefore, we propose a segmentation algorithm for tongue images collected in complex multi-device and multi-user environments. We use convolutional attention and extend state space models to the 2D environment in the encoder. Then, cross-layer connection fusion is used in the decoder part to fuse shallow texture and deep semantic features. Through segmentation experiments on tongue image datasets collected by patients and doctors in real-world settings, our algorithm significantly improves segmentation performance and accuracy.

Development and Application of Traditional Chinese Medicine Using AI Machine Learning and Deep Learning Strategies

Traditional Chinese medicine (TCM) has been used for thousands of years and has been proven to be effective at treating many complicated illnesses with minimal side effects. The application and advancement of TCM are, however, constrained by the absence of objective measuring standards due to its relatively abstract diagnostic methods and syndrome differentiation theories. Ongoing developments in machine learning (ML) and deep learning (DL), specifically in computer vision (CV) and natural language processing (NLP), offer novel opportunities to modernize TCM by exploring the profound connotations of its theory. This review begins with an overview of the ML and DL methods employed in TCM; this is followed by practical instances of these applications. Furthermore, extensive discussions emphasize the mature integration of ML and DL in TCM, such as tongue diagnosis, pulse diagnosis, and syndrome differentiation treatment, highlighting their early successful application in the TCM field. Finally, this study validates the accomplishments and addresses the problems and challenges posed by the application and development of TCM powered by ML and DL. As ML and DL techniques continue to evolve, modern technology will spark new advances in TCM.

Application of intelligent tongue image analysis in Conjunction with microbiomes in the diagnosis of MAFLD

Background

Metabolic associated fatty liver disease (MAFLD) is a widespread liver disease that can lead to liver fibrosis and cirrhosis. Therefore, it is essential to develop early diagnosic and screening methods.

Methods

We performed a cross-sectional observational study. In this study, based on data from 92 patients with MAFLD and 74 healthy individuals, we observed the characteristics of tongue images, tongue coating and intestinal flora. A generative adversarial network was used to extract tongue image features, and 16S rRNA sequencing was performed using the tongue coating and intestinal flora. We then applied tongue image analysis technology combined with microbiome technology to obtain an MAFLD early screening model with higher accuracy. In addition, we compared different modelling methods, including Extreme Gradient Boosting (XGBoost), random forest, neural networks(MLP), stochastic gradient descent(SGD), and support vector machine(SVM).

Results

The results show that tongue-coating Streptococcus and Rothia, intestinal Blautia, and Streptococcus are potential biomarkers for MAFLD. The diagnostic model jointly incorporating tongue image features, basic information (gender, age, BMI), and tongue coating marker flora (Streptococcus, Rothia), can have an accuracy of 96.39%, higher than the accuracy value except for bacteria.

Conclusion

Combining computer-intelligent tongue diagnosis with microbiome technology enhances MAFLD diagnostic accuracy and provides a convenient early screening reference.

A review of traditional Chinese medicine diagnosis using machine learning: Inspection, auscultation-olfaction, inquiry, and palpation

Traditional Chinese medicine (TCM) is an essential part of the Chinese medical system and is recognized by the World Health Organization as an important alternative medicine. As an important part of TCM, TCM diagnosis is a method to understand a patient's illness, analyze its state, and identify syndromes. In the long-term clinical diagnosis practice of TCM, four fundamental and effective diagnostic methods of inspection, auscultation-olfaction, inquiry, and palpation (IAOIP) have been formed. However, the diagnostic information in TCM is diverse, and the diagnostic process depends on doctors' experience, which is subject to a high-level subjectivity. At present, the research on the automated diagnosis of TCM based on machine learning is booming. Machine learning, which includes deep learning, is an essential part of artificial intelligence (AI), which provides new ideas for the objective and AI-related research of TCM. This paper aims to review and summarize the current research status of machine learning in TCM diagnosis. First, we review some key factors for the application of machine learning in TCM diagnosis, including data, data preprocessing, machine learning models, and evaluation metrics. Second, we review and summarize the research and applications of machine learning methods in TCM IAOIP and the synthesis of the four diagnostic methods. Finally, we discuss the challenges and research directions of using machine learning methods for TCM diagnosis.

Noninvasive Diagnostic Technique for Nonalcoholic Fatty Liver Disease Based on Features of Tongue Images

OBJECTIVE

To investigate a new noninvasive diagnostic model for nonalcoholic fatty liver disease (NAFLD) based on features of tongue images.

METHODS

Healthy controls and volunteers confirmed to have NAFLD by liver ultrasound were recruited from China-Japan Friendship Hospital between September 2018 and May 2019, then the anthropometric indexes and sampled tongue images were measured. The tongue images were labeled by features, based on a brief protocol, without knowing any other clinical data, after a series of corrections and data cleaning. The algorithm was trained on images using labels and several anthropometric indexes for inputs, utilizing machine learning technology. Finally, a logistic regression algorithm and a decision tree model were constructed as 2 diagnostic models for NAFLD.

RESULTS

A total of 720 subjects were enrolled in this study, including 432 patients with NAFLD and 288 healthy volunteers. Of them, 482 were randomly allocated into the training set and 238 into the validation set. The diagnostic model based on logistic regression exhibited excellent performance: in validation set, it achieved an accuracy of 86.98%, sensitivity of 91.43%, and specificity of 80.61%; with an area under the curve (AUC) of 0.93 [95% confidence interval (CI) 0.68-0.98]. The decision tree model achieved an accuracy of 81.09%, sensitivity of 91.43%, and specificity of 66.33%; with an AUC of 0.89 (95% CI 0.66-0.92) in validation set.

CONCLUSIONS

The features of tongue images were associated with NAFLD. Both the 2 diagnostic models, which would be convenient, noninvasive, lightweight, rapid, and inexpensive technical references for early screening, can accurately distinguish NAFLD and are worth further study.

Accurate estimation of biological age and its application in disease prediction using a multimodal image Transformer system

Aging in an individual refers to the temporal change, mostly decline, in the body's ability to meet physiological demands. Biological age (BA) is a biomarker of chronological aging and can be used to stratify populations to predict certain age-related chronic diseases. BA can be predicted from biomedical features such as brain MRI, retinal, or facial images, but the inherent heterogeneity in the aging process limits the usefulness of BA predicted from individual body systems. In this paper, we developed a multimodal Transformer-based architecture with cross-attention which was able to combine facial, tongue, and retinal images to estimate BA. We trained our model using facial, tongue, and retinal images from 11,223 healthy subjects and demonstrated that using a fusion of the three image modalities achieved the most accurate BA predictions. We validated our approach on a test population of 2,840 individuals with six chronic diseases and obtained significant difference between chronological age and BA (AgeDiff) than that of healthy subjects. We showed that AgeDiff has the potential to be utilized as a standalone biomarker or conjunctively alongside other known factors for risk stratification and progression prediction of chronic diseases. Our results therefore highlight the feasibility of using multimodal images to estimate and interrogate the aging process.

In-depth metaproteomics analysis of tongue coating for gastric cancer: a multicenter diagnostic research study

BACKGROUND

Our previous study revealed marked differences in tongue images between individuals with gastric cancer and those without gastric cancer. However, the biological mechanism of tongue images as a disease indicator remains unclear. Tongue coating, a major factor in tongue appearance, is the visible layer on the tongue dorsum that provides a vital environment for oral microorganisms. While oral microorganisms are associated with gastric and intestinal diseases, the comprehensive function profiles of oral microbiota remain incompletely understood. Metaproteomics has unique strength in revealing functional profiles of microbiota that aid in comprehending the mechanism behind specific tongue coating formation and its role as an indicator of gastric cancer.

METHODS

We employed pressure cycling technology and data-independent acquisition (PCT-DIA) mass spectrometry to extract and identify tongue-coating proteins from 180 gastric cancer patients and 185 non-gastric cancer patients across 5 independent research centers in China. Additionally, we investigated the temporal stability of tongue-coating proteins based on a time-series cohort. Finally, we constructed a machine learning model using the stochastic gradient boosting algorithm to identify individuals at high risk of gastric cancer based on tongue-coating microbial proteins.

RESULTS

We measured 1432 human-derived proteins and 13,780 microbial proteins from 345 tongue-coating samples. The abundance of tongue-coating proteins exhibited high temporal stability within an individual. Notably, we observed the downregulation of human keratins KRT2 and KRT9 on the tongue surface, as well as the downregulation of ABC transporter COG1136 in microbiota, in gastric cancer patients. This suggests a decline in the defense capacity of the lingual mucosa. Finally, we established a machine learning model that employs 50 microbial proteins of tongue coating to identify individuals at a high risk of gastric cancer, achieving an area under the curve (AUC) of 0.91 in the independent validation cohort.

CONCLUSIONS

We characterized the alterations in tongue-coating proteins among gastric cancer patients and constructed a gastric cancer screening model based on microbial-derived tongue-coating proteins. Tongue-coating proteins are shown as a promising indicator for identifying high-risk groups for gastric cancer. Video Abstract.

Study on the difference and regularity of tongue images in 309 patients with different pathological stages of non-small cell lung cancer

BACKGROUND

Tongue diagnosis is a crucial traditional Chinese medicine (TCM) inspection method for TCM syndrome differentiation and treatment.

OBJECTIVE

The primary research focus was on tongue image characteristic parameters of patients with non-small cell lung cancer (NSCLC). Analysis of the tongue image parameters of various pathological stages of NSCLC provides technical support for establishing an integrated Chinese and Western auxiliary diagnosis and efficacy evaluation medicine system for lung cancer that integrates tongue image features.

METHODS

Tongue image characteristics of 309 patients with NSCLC and 206 controls were collected and analyzed clinically. The T-test or rank sum test and logistic regression analysis were applied to analyze the characteristics of tongue image indicators of different pathological stages of NSCLC.

RESULTS

There were differences in tongue image characteristics in the NSCLC group compared to the control group. The tongue quality and brightness of the tongue coating in the NSCLC group increased, the red component was reduced, the tongue coating thickened, and the yellow component increased compared to the healthy control group. A comparison of tongue image indexes of NSCLC in different pathological stages showed that stage IV had lower TB-b and higher TB-a than stage I. In addition, stage IV had lower TB-b than stage II + III, showing an increase in the blue and red components of the tongue in stage IV and the appearance of cyanotic tongue features.

CONCLUSION

The tongue image characteristics of NSCLC patients differed from those of the control group. Tongue imaging indicators can reflect the characteristics of tongue images of patients with NSCLC. The tongue image characteristics of patients with stage IV lung cancer are bluish and purple compared with those with stage I, II, and III. It is suggested that the tongue's image characteristics can be used as a reference for the pathological classification of NSCLC and judgment of the disease process.

Core characteristics of sublingual veins analysis and its relationship with hypertension

BACKGROUND

The sublingual vein (SV) is a specialized diagnostic method used in Traditional Chinese Medicine (TCM). Despite its ability to objectively reflect blood flow, SV is often overlooked in clinical practice.

OBJECTIVE

This study aims to analyze the core characteristics of SV and investigate the in-depth relationship between its digital characteristics and hypertension. The goal is to find a link between SV and hypertension and break out of the current situation.

METHODS

Modern digital analysis techniques were applied to the traditional SV diagnostic theory. In a controlled study with 204 participants, the digital characteristics of SV were documented using TFDA-1, and its color value was analyzed using TDAS. Morphological characteristics of SV, such as trunklength, width, and tortuosity, were examined by combining computer vision with expert interpretation. This involved the application of automatic ranging methods and a rectangular approximation algorithm, which are novel approaches in the field of TCM. The t-test and Mann-Whitney U test were used to analyze the digital characteristics of SV in hypertension. Binary logistic regression and neural network models were established using machine learning to explore the deep relationship between SV characteristics and hypertension.

RESULTS

There was a significant difference of the tortuosity of SV between the two groups (Z=-2.629, p= 0.009). The results revealed thick width of SV (OR = 2.64, 95% CI: 1.02-6.79) was the risk factor for hypertension. Addition of SV characteristics improved overall percent correct for hypertension prediction to 80%.

CONCLUSION

TCM method of diagnosis of SV has been greatly expanded in terms of technical means, and the close relationship between SV and hypertension has been found in clinical data.

ODQN-Net: Optimized Deep Q Neural Networks for Disease Prediction Through Tongue Image Analysis Using Remora Optimization Algorithm

Tongue analysis plays the major role in disease type prediction and classification according to Indian ayurvedic medicine. Traditionally, there is a manual inspection of tongue image by the expert ayurvedic doctor to identify or predict the disease. However, this is time-consuming and even imprecise. Due to the advancements in recent machine learning models, several researchers addressed the disease prediction from tongue image analysis. However, they have failed to provide enough accuracy. In addition, multiclass disease classification with enhanced accuracy is still a challenging task. Therefore, this article focuses on the development of optimized deep q-neural network (DQNN) for disease identification and classification from tongue images, hereafter referred as ODQN-Net. Initially, the multiscale retinex approach is introduced for enhancing the quality of tongue images, which also acts as a noise removal technique. In addition, a local ternary pattern is used to extract the disease-specific and disease-dependent features based on color analysis. Then, the best features are extracted from the available features set using the natural inspired Remora optimization algorithm with reduced computational time. Finally, the DQNN model is used to classify the type of diseases from these pretrained features. The obtained simulation performance on tongue imaging data set proved that the proposed ODQN-Net resulted in superior performance compared with state-of-the-art approaches with 99.17% of accuracy and 99.75% and 99.84% of F1-score and Mathew's correlation coefficient, respectively.

Machine learning prediction models for different stages of non-small cell lung cancer based on tongue and tumor marker: a pilot study

OBJECTIVE

To analyze the tongue feature of NSCLC at different stages, as well as the correlation between tongue feature and tumor marker, and investigate the feasibility of establishing prediction models for NSCLC at different stages based on tongue feature and tumor marker.

METHODS

Tongue images were collected from non-advanced NSCLC patients (n = 109) and advanced NSCLC patients (n = 110), analyzed the tongue images to obtain tongue feature, and analyzed the correlation between tongue feature and tumor marker in different stages of NSCLC. On this basis, six classifiers, decision tree, logistic regression, SVM, random forest, naive bayes, and neural network, were used to establish prediction models for different stages of NSCLC based on tongue feature and tumor marker.

RESULTS

There were statistically significant differences in tongue feature between the non-advanced and advanced NSCLC groups. In the advanced NSCLC group, the number of indexes with statistically significant correlations between tongue feature and tumor marker was significantly higher than in the non-advanced NSCLC group, and the correlations were stronger. Support Vector Machine (SVM), decision tree, and logistic regression among the machine learning methods performed poorly in models with different stages of NSCLC. Neural network, random forest and naive bayes had better classification efficiency for the data set of tongue feature and tumor marker and baseline. The models' classification accuracies were 0.767 ± 0.081, 0.718 ± 0.062, and 0.688 ± 0.070, respectively, and the AUCs were 0.793 ± 0.086, 0.779 ± 0.075, and 0.771 ± 0.072, respectively.

CONCLUSIONS

There were statistically significant differences in tongue feature between different stages of NSCLC, with advanced NSCLC tongue feature being more closely correlated with tumor marker. Due to the limited information, single data sources including baseline, tongue feature, and tumor marker cannot be used to identify the different stages of NSCLC in this pilot study. In addition to the logistic regression method, other machine learning methods, based on tumor marker and baseline data sets, can effectively improve the differential diagnosis efficiency of different stages of NSCLC by adding tongue image data, which requires further verification based on large sample studies in the future.

Relationship chains of subhealth physical examination indicators: a cross-sectional study using the PLS-SEM approach

Subhealth is a transitional state between health and disease, and it can be detected through routine physical check-ups. However, the complexity and diversity of physical examination items and the difficulty of quantifying subhealth manifestations are the main problems that hinder its treatment. The aim of this study was to systematically investigate the physical examination performance of the subhealthy population and further explore the deeper relationships between indicators. Indicators were obtained for 878 subjects, including basic information, Western medicine indicators, inquiries of traditional Chinese medicine and sublingual vein (SV) characteristics. Statistical differences were analysed using R software. To explore the distribution of symptoms and symptom clusters in subhealth, partial least squares-structural equation modelling (PLS-SEM) was applied to the subhealth physical examination index, and a structural model was developed to verify whether the relationship chain between the latent variables was reasonable. Finally, the reliability and validity of the PLS-SE model were assessed. The most common subclinical clinical symptoms were limb soreness (37.6%), fatigue (31.6%), shoulder and neck pain (30.5%) and dry eyes (29.2%). The redness of the SV in the subhealthy group was paler than that in the healthy group (p < 0.001). This study validates the establishment of the directed acyclic relationship chain in the subhealthy group: the path from routine blood tests to lipid metabolism (t = 7.878, p < 0.001), the path from lipid metabolism to obesity (t = 8.410, p < 0.001), the path from obesity to SV characteristics (t = 2.237, p = 0.025), and the path from liver function to SV characteristics (t = 2.215, p = 0.027). The innovative application of PLS-SEM to the study of subhealth has revealed the existence of a chain of relationships between physical examination indicators, which will provide a basis for further exploration of subhealth mechanisms and causal inference. This study has identified the typical symptoms of subhealth, and their early management will help to advance the treatment of diseases.

A lung cancer risk warning model based on tongue images

Objective: To investigate the tongue image features of patients with lung cancer and benign pulmonary nodules and to construct a lung cancer risk warning model using machine learning methods. Methods: From July 2020 to March 2022, we collected 862 participants including 263 patients with lung cancer, 292 patients with benign pulmonary nodules, and 307 healthy subjects. The TFDA-1 digital tongue diagnosis instrument was used to capture tongue images, using feature extraction technology to obtain the index of the tongue images. The statistical characteristics and correlations of the tongue index were analyzed, and six machine learning algorithms were used to build prediction models of lung cancer based on different data sets. Results: Patients with benign pulmonary nodules had different statistical characteristics and correlations of tongue image data than patients with lung cancer. Among the models based on tongue image data, the random forest prediction model performed the best, with a model accuracy of 0.679 ± 0.048 and an AUC of 0.752 ± 0.051. The accuracy for the logistic regression, decision tree, SVM, random forest, neural network, and naïve bayes models based on both the baseline and tongue image data were 0.760 ± 0.021, 0.764 ± 0.043, 0.774 ± 0.029, 0.770 ± 0.050, 0.762 ± 0.059, and 0.709 ± 0.052, respectively, while the corresponding AUCs were 0.808 ± 0.031, 0.764 ± 0.033, 0.755 ± 0.027, 0.804 ± 0.029, 0.777 ± 0.044, and 0.795 ± 0.039, respectively. Conclusion: The tongue diagnosis data under the guidance of traditional Chinese medicine diagnostic theory was useful. The performance of models built on tongue image and baseline data was superior to that of the models built using only the tongue image data or the baseline data. Adding objective tongue image data to baseline data can significantly improve the efficacy of lung cancer prediction models.

Construction of Tongue Image-Based Machine Learning Model for Screening Patients with Gastric Precancerous Lesions

Screening patients with precancerous lesions of gastric cancer (PLGC) is important for gastric cancer prevention. The accuracy and convenience of PLGC screening could be improved with the use of machine learning methodologies to uncover and integrate valuable characteristics of noninvasive medical images related to PLGC. In this study, we therefore focused on tongue images and for the first time constructed a tongue image-based PLGC screening deep learning model (AITongue). The AITongue model uncovered potential associations between tongue image characteristics and PLGC, and integrated canonical risk factors, including age, sex, and Hp infection. Five-fold cross validation analysis on an independent cohort of 1995 patients revealed the AITongue model could screen PLGC individuals with an AUC of 0.75, 10.3% higher than that of the model with only including canonical risk factors. Of note, we investigated the value of the AITongue model in predicting PLGC risk by establishing a prospective PLGC follow-up cohort, reaching an AUC of 0.71. In addition, we developed a smartphone-based app screening system to enhance the application convenience of the AITongue model in the natural population from high-risk areas of gastric cancer in China. Collectively, our study has demonstrated the value of tongue image characteristics in PLGC screening and risk prediction.

A new model for predicting the occurrence of polycystic ovary syndrome: Based on data of tongue and pulse

Background and objective

Polycystic ovary syndrome is one of the most common types of endocrine and metabolic diseases in women of reproductive age that needs to be screened early and assessed non-invasively. The objective of the current study was to develop prediction models for polycystic ovary syndrome based on data of tongue and pulse using machine learning techniques.

Methods

A dataset of 285 polycystic ovary syndrome patients and 201 healthy women were investigated to identify the significant tongue and pulse parameters for predicting polycystic ovary syndrome. In this study, feature selection was performed using least absolute shrinkage and selection operator regression. Several machine learning algorithms (multilayer perceptron classifier, eXtreme gradient boosting classifier, and support vector machine) were used to construct the classification models to predict the presence of polycystic ovary syndrome.

Results

TB-L, TB-a, TB-b, TC-L, TC-a, h3, and h4/h1 in tongue and pulse parameters were statistically associated with polycystic ovary syndrome presence. Among the several machine learning techniques, the support vector machine model was optimal for the comprehensive evaluation of this dataset and deduced the area under the receiver operating characteristic curve, DeLong test, calibration curve, and decision curve analysis.

Conclusion

The machine learning model with tongue and pulse factors can predict the existence of polycystic ovary syndrome precisely.

A survey of artificial intelligence in tongue image for disease diagnosis and syndrome differentiation

The rapid development of artificial intelligence technology has gradually extended from the general field to all walks of life, and intelligent tongue diagnosis is the product of a miraculous connection between this new discipline and traditional disciplines. We reviewed the deep learning methods and machine learning applied in tongue image analysis that have been studied in the last 5 years, focusing on tongue image calibration, detection, segmentation, and classification of diseases, syndromes, and symptoms/signs. Introducing technical evolutions or emerging technologies were applied in tongue image analysis; as we have noticed, attention mechanism, multiscale features, and prior knowledge were successfully applied in it, and we emphasized the value of combining deep learning with traditional methods. We also pointed out two major problems concerned with data set construction and the low reliability of performance evaluation that exist in this field based on the basic essence of tongue diagnosis in traditional Chinese medicine. Finally, a perspective on the future of intelligent tongue diagnosis was presented; we believe that the self-supervised method, multimodal information fusion, and the study of tongue pathology will have great research significance.

An Intelligent Tongue Diagnosis System via Deep Learning on the Android Platform

To quickly and accurately identify the pathological features of the tongue, we developed an intelligent tongue diagnosis system that uses deep learning on a mobile terminal. We also propose an efficient and accurate tongue image processing algorithm framework to infer the category of the tongue. First, a software system integrating registration, login, account management, tongue image recognition, and doctor-patient dialogue was developed based on the Android platform. Then, the deep learning models, based on the official benchmark models, were trained by using the tongue image datasets. The tongue diagnosis algorithm framework includes the YOLOv5s6, U-Net, and MobileNetV3 networks, which are employed for tongue recognition, tongue region segmentation, and tongue feature classification (tooth marks, spots, and fissures), respectively. The experimental results demonstrate that the performance of the tongue diagnosis model was satisfying, and the accuracy of the final classification of tooth marks, spots, and fissures was 93.33%, 89.60%, and 97.67%, respectively. The construction of this system has a certain reference value for the objectification and intelligence of tongue diagnosis.

Deep Learning Multi-label Tongue Image Analysis and Its Application in a Population Undergoing Routine Medical Checkup

Background

Research on intelligent tongue diagnosis is a main direction in the modernization of tongue diagnosis technology. Identification of tongue shape and texture features is a difficult task for tongue diagnosis in traditional Chinese medicine (TCM). This study aimed to explore the application of deep learning techniques in tongue image analyses.

Methods

A total of 8676 tongue images were annotated by clinical experts, into seven categories, including the fissured tongue, tooth-marked tongue, stasis tongue, spotted tongue, greasy coating, peeled coating, and rotten coating. Based on the labeled tongue images, the deep learning model faster region-based convolutional neural networks (Faster R-CNN) was utilized to classify tongue images. Four performance indices, i.e., accuracy, recall, precision, and F1-score, were selected to evaluate the model. Also, we applied it to analyze tongue image features of 3601 medical checkup participants in order to explore gender and age factors and the correlations among tongue features in diseases through complex networks.

Results

The average accuracy, recall, precision, and F1-score of our model achieved 90.67%, 91.25%, 99.28%, and 95.00%, respectively. Over the tongue images from the medical checkup population, the model Faster R-CNN detected 41.49% fissured tongue images, 37.16% tooth-marked tongue images, 29.66% greasy coating images, 18.66% spotted tongue images, 9.97% stasis tongue images, 3.97% peeled coating images, and 1.22% rotten coating images. There were significant differences in the incidence of the fissured tongue, tooth-marked tongue, spotted tongue, and greasy coating among age and gender. Complex networks revealed that fissured tongue and tooth-marked were closely related to hypertension, dyslipidemia, overweight and nonalcoholic fatty liver disease (NAFLD), and a greasy coating tongue was associated with hypertension and overweight.

Conclusion

The model Faster R-CNN shows good performance in the tongue image classification. And we have preliminarily revealed the relationship between tongue features and gender, age, and metabolic diseases in a medical checkup population.

Development and validation of a neural network for NAFLD diagnosis

Non-Alcoholic Fatty Liver Disease (NAFLD) affects about 20–30% of the adult population in developed countries and is an increasingly important cause of hepatocellular carcinoma. Liver ultrasound (US) is widely used as a noninvasive method to diagnose NAFLD. However, the intensive use of US is not cost-effective and increases the burden on the healthcare system. Electronic medical records facilitate large-scale epidemiological studies and, existing NAFLD scores often require clinical and anthropometric parameters that may not be captured in those databases. Our goal was to develop and validate a simple Neural Network (NN)-based web app that could be used to predict NAFLD particularly its absence. The study included 2970 subjects; training and testing of the neural network using a train–test-split approach was done on 2869 of them. From another population consisting of 2301 subjects, a further 100 subjects were randomly extracted to test the web app. A search was made to find the best parameters for the NN and then this NN was exported for incorporation into a local web app. The percentage of accuracy, area under the ROC curve, confusion matrix, Positive (PPV) and Negative Predicted Value (NPV) values, precision, recall and f1-score were verified. After that, Explainability (XAI) was analyzed to understand the diagnostic reasoning of the NN. Finally, in the local web app, the specificity and sensitivity values were checked. The NN achieved a percentage of accuracy during testing of 77.0%, with an area under the ROC curve value of 0.82. Thus, in the web app the NN evidenced to achieve good results, with a specificity of 1.00 and sensitivity of 0.73. The described approach can be used to support NAFLD diagnosis, reducing healthcare costs. The NN-based web app is easy to apply and the required parameters are easily found in healthcare databases.