Clinical Validation of a Deep Learning-Based Hybrid (Greulich-Pyle and Modified Tanner-Whitehouse) Method for Bone Age Assessment

Olecranon bone age assessment in puberty using a lateral elbow radiograph and a deep-learning model

OBJECTIVES

To improve pubertal bone age (BA) evaluation by developing a precise and practical elbow BA classification using the olecranon, and a deep-learning AI model.

MATERIALS AND METHODS

Lateral elbow radiographs taken for BA evaluation in children under 18 years were collected from January 2020 to June 2022, retrospectively. A novel classification and the olecranon BA were established based on the morphological changes in the olecranon ossification process during puberty. The olecranon BA was compared with other elbow and hand BA methods, using intraclass correlation coefficients (ICCs), and a deep-learning AI model was developed.

RESULTS

A total of 3508 lateral elbow radiographs (mean age 9.8 ± 1.8 years) were collected. The olecranon BA showed the highest applicability (100%) and interobserver agreement (ICC 0.993) among elbow BA methods. It showed excellent reliability with Sauvegrain (0.967 in girls, 0.969 in boys) and Dimeglio (0.978 in girls, 0.978 in boys) elbow BA methods, as well as Korean standard (KS) hand BA in boys (0.917), and good reliability with KS in girls (0.896) and Greulich-Pyle (GP)/Tanner-Whitehouse (TW)3 (0.835 in girls, 0.895 in boys) hand BA methods. The AI model for olecranon BA showed an accuracy of 0.96 and a specificity of 0.98 with EfficientDet-b4. External validation showed an accuracy of 0.86 and a specificity of 0.91.

CONCLUSION

The olecranon BA evaluation for puberty, requiring only a lateral elbow radiograph, showed the highest applicability and interobserver agreement, and excellent reliability with other BA evaluation methods, along with a high performance of the AI model.

CLINICAL RELEVANCE STATEMENT

This AI model uses a single lateral elbow radiograph to determine bone age for puberty from the olecranon ossification center and can improve pubertal bone age assessment with the highest applicability and excellent reliability compared to previous methods.

KEY POINTS

Elbow bone age is valuable for pubertal bone age assessment, but conventional methods have limitations. Olecranon bone age and its AI model showed high performances for pubertal bone age assessment. Olecranon bone age system is practical and accurate while requiring only a single lateral elbow radiograph.

Assessment of Bone Age Based on Hand Radiographs Using Regression-Based Multi-Modal Deep Learning

(1) Objective: In this study, a regression-based multi-modal deep learning model was developed for use in bone age assessment (BAA) utilizing hand radiographic images and clinical data, including patient gender and chronological age, as input data. (2) Methods: A dataset of hand radiographic images from 2974 pediatric patients was used to develop a regression-based multi-modal BAA model. This model integrates hand radiographs using EfficientNetV2S convolutional neural networks (CNNs) and clinical data (gender and chronological age) processed by a simple deep neural network (DNN). This approach enhances the model's robustness and diagnostic precision, addressing challenges related to imbalanced data distribution and limited sample sizes. (3) Results: The model exhibited good performance on BAA, with an overall mean absolute error (MAE) of 0.410, root mean square error (RMSE) of 0.637, and accuracy of 91.1%. Subgroup analysis revealed higher accuracy in females ≤ 11 years (MAE: 0.267, RMSE: 0.453, accuracy: 95.0%) and >11 years (MAE: 0.402, RMSE: 0.634, accuracy 92.4%) compared to males ≤ 13 years (MAE: 0.665, RMSE: 0.912, accuracy: 79.7%) and >13 years (MAE: 0.647, RMSE: 1.302, accuracy: 84.6%). (4) Conclusion: This model showed a generally good performance on BAA, showing a better performance in female pediatrics compared to male pediatrics and an especially robust performance in female pediatrics ≤ 11 years.

Artificial intelligence model system for bone age assessment of preschool children

BACKGROUD

Our study aimed to assess the impact of inter- and intra-observer variations when utilizing an artificial intelligence (AI) system for bone age assessment (BAA) of preschool children.

METHODS

A retrospective study was conducted involving a total sample of 53 female individuals and 41 male individuals aged 3-6 years in China. Radiographs were assessed by four mid-level radiology reviewers using the TW3 and RUS-CHN methods. Bone age (BA) was analyzed in two separate situations, with/without the assistance of AI. Following a 4-week wash-out period, radiographs were reevaluated in the same manner. Accuracy metrics, the correlation coefficient (ICC)and Bland-Altman plots were employed.

RESULTS

The accuracy of BAA by the reviewers was significantly improved with AI. The results of RMSE and MAE decreased in both methods (p < 0.001). When comparing inter-observer agreement in both methods and intra-observer reproducibility in two interpretations, the ICC results were improved with AI. The ICC values increased in both two interpretations for both methods and exceeded 0.99 with AI.

CONCLUSION

In the assessment of BA for preschool children, AI was found to be capable of reducing inter-observer variability and enhancing intra-observer reproducibility, which can be considered an important tool for clinical work by radiologists.

IMPACT

The RUS-CHN method is a special bone age method devised to be suitable for Chinese children. The preschool stage is a critical phase for children, marked by a high degree of variability that renders BA prediction challenging. The accuracy of BAA by the reviewers can be significantly improved with the aid of an AI model system. This study is the first to assess the impact of inter- and intra-observer variations when utilizing an AI model system for BAA of preschool children using both the TW3 and RUS-CHN methods.

Survey on Value Elements Provided by Artificial Intelligence and Their Eligibility for Insurance Coverage With an Emphasis on Patient-Centered Outcomes

OBJECTIVE

This study aims to explore the opinions on the insurance coverage of artificial intelligence (AI), as categorized based on the distinct value elements offered by AI, with a specific focus on patient-centered outcomes (PCOs). PCOs are distinguished from traditional clinical outcomes and focus on patient-reported experiences and values such as quality of life, functionality, well-being, physical or emotional status, and convenience.

MATERIALS AND METHODS

We classified the value elements provided by AI into four dimensions: clinical outcomes, economic aspects, organizational aspects, and non-clinical PCOs. The survey comprised three sections: 1) experiences with PCOs in evaluating AI, 2) opinions on the coverage of AI by the National Health Insurance of the Republic of Korea when AI demonstrated benefits across the four value elements, and 3) respondent characteristics. The opinions regarding AI insurance coverage were assessed dichotomously and semi-quantitatively: non-approval (0) vs. approval (on a 1-10 weight scale, with 10 indicating the strongest approval). The survey was conducted from July 4 to 26, 2023, using a web-based method. Responses to PCOs and other value elements were compared.

RESULTS

Among 200 respondents, 44 (22%) were patients/patient representatives, 64 (32%) were industry/developers, 60 (30%) were medical practitioners/doctors, and 32 (16%) were government health personnel. The level of experience with PCOs regarding AI was low, with only 7% (14/200) having direct experience and 10% (20/200) having any experience (either direct or indirect). The approval rate for insurance coverage for PCOs was 74% (148/200), significantly lower than the corresponding rates for other value elements (82.5%-93.5%; P ≤ 0.034). The approval strength was significantly lower for PCOs, with a mean weight ± standard deviation of 5.1 ± 3.5, compared to other value elements (P ≤ 0.036).

CONCLUSION

There is currently limited demand for insurance coverage for AI that demonstrates benefits in terms of non-clinical PCOs.

Use of artificial intelligence in determination of bone age of the healthy individuals: A scoping review

BACKGROUND

Bone age assessment, as an indicator of biological age, is widely used in orthodontics and pediatric endocrinology. Owing to significant subject variations in the manual method of assessment, artificial intelligence (AI), machine learning (ML), and deep learning (DL) play a significant role in this aspect. A scoping review was conducted to search the existing literature on the role of AI, ML, and DL in skeletal age or bone age assessment in healthy individuals.

METHODS

A literature search was conducted in PubMed, Scopus, and Web of Science from January 2012 to December 2022 using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses-Extension for Scoping Reviews (PRISMA-ScR) and Joanna Briggs Institute guidelines. Grey literature was searched using Google Scholar and OpenGrey. Hand-searching of the articles in all the reputed orthodontic journals and the references of the included articles were also searched for relevant articles for the present scoping review.

RESULTS

Nineteen articles that fulfilled the inclusion criteria were included. Ten studies used skeletal maturity indicators based on hand and wrist radiographs, two studies used magnetic resonance imaging and seven studies used cervical vertebrae maturity indicators based on lateral cephalograms to assess the skeletal age of the individuals. Most of these studies were published in non-orthodontic medical journals. BoneXpert automated software was the most commonly used software, followed by DL models, and ML models in the studies for assessment of bone age. The automated method was found to be as reliable as the manual method for assessment.

CONCLUSIONS

This scoping review validated the use of AI, ML, or DL in bone age assessment of individuals. A more uniform distribution of sufficient samples in different stages of maturation, use of three-dimensional inputs such as magnetic resonance imaging, and cone beam computed tomography is required for better training of the models to generalize the outputs for use in the target population.

An artificial intelligence-based bone age assessment model for Han and Tibetan children

Background: Manual bone age assessment (BAA) is associated with longer interpretation time and higher cost and variability, thus posing challenges in areas with restricted medical facilities, such as the high-altitude Tibetan Plateau. The application of artificial intelligence (AI) for automating BAA could facilitate resolving this issue. This study aimed to develop an AI-based BAA model for Han and Tibetan children. Methods: A model named "EVG-BANet" was trained using three datasets, including the Radiology Society of North America (RSNA) dataset (training set n = 12611, validation set n = 1425, and test set n = 200), the Radiological Hand Pose Estimation (RHPE) dataset (training set n = 5491, validation set n = 713, and test set n = 79), and a self-established local dataset [training set n = 825 and test set n = 351 (Han n = 216 and Tibetan n = 135)]. An open-access state-of-the-art model BoNet was used for comparison. The accuracy and generalizability of the two models were evaluated using the abovementioned three test sets and an external test set (n = 256, all were Tibetan). Mean absolute difference (MAD) and accuracy within 1 year were used as indicators. Bias was evaluated by comparing the MAD between the demographic groups. Results: EVG-BANet outperformed BoNet in the MAD on the RHPE test set (0.52 vs. 0.63 years, p < 0.001), the local test set (0.47 vs. 0.62 years, p < 0.001), and the external test set (0.53 vs. 0.66 years, p < 0.001) and exhibited a comparable MAD on the RSNA test set (0.34 vs. 0.35 years, p = 0.934). EVG-BANet achieved accuracy within 1 year of 97.7% on the local test set (BoNet 90%, p < 0.001) and 89.5% on the external test set (BoNet 85.5%, p = 0.066). EVG-BANet showed no bias in the local test set but exhibited a bias related to chronological age in the external test set. Conclusion: EVG-BANet can accurately predict the bone age (BA) for both Han children and Tibetan children living in the Tibetan Plateau with limited healthcare facilities.

Region fine-grained attention network for accurate bone age assessment

Bone age assessment plays a vital role in monitoring the growth and development of adolescents. However, it is still challenging to obtain precise bone age from hand radiography due to these problems: 1) Hand bone varies greatly and is always masked by the background; 2) the hand bone radiographs with successive ages offer high similarity. To solve such issues, a region fine-grained attention network (RFGA-Net) was proposed for bone age assessment, where the region aware attention (RAA) module was developed to distinguish the skeletal regions from the background by modeling global spatial dependency; then the fine-grained feature attention (FFA) module was devised to identify similar bone radiographs by recognizing critical fine-grained feature regions. The experimental results demonstrate that the proposed RFGA-Net shows the best performance on the Radiological Society of North America (RSNA) pediatric bone dataset, achieving the mean absolute error (MAE) of 3.34 and the root mean square error (RMSE) of 4.02, respectively.

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia

OBJECTIVE

We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit.

MATERIALS AND METHODS

A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020.

RESULTS

Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020.

CONCLUSION

The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

Bone Age Assessment Using Artificial Intelligence in Korean Pediatric Population: A Comparison of Deep-Learning Models Trained With Healthy Chronological and Greulich-Pyle Ages as Labels

OBJECTIVE

To develop a deep-learning-based bone age prediction model optimized for Korean children and adolescents and evaluate its feasibility by comparing it with a Greulich-Pyle-based deep-learning model.

MATERIALS AND METHODS

A convolutional neural network was trained to predict age according to the bone development shown on a hand radiograph (bone age) using 21036 hand radiographs of Korean children and adolescents without known bone development-affecting diseases/conditions obtained between 1998 and 2019 (median age [interquartile range {IQR}], 9 [7-12] years; male:female, 11794:9242) and their chronological ages as labels (Korean model). We constructed 2 separate external datasets consisting of Korean children and adolescents with healthy bone development (Institution 1: n = 343; median age [IQR], 10 [4-15] years; male: female, 183:160; Institution 2: n = 321; median age [IQR], 9 [5-14] years; male: female, 164:157) to test the model performance. The mean absolute error (MAE), root mean square error (RMSE), and proportions of bone age predictions within 6, 12, 18, and 24 months of the reference age (chronological age) were compared between the Korean model and a commercial model (VUNO Med-BoneAge version 1.1; VUNO) trained with Greulich-Pyle-based age as the label (GP-based model).

RESULTS

Compared with the GP-based model, the Korean model showed a lower RMSE (11.2 vs. 13.8 months; P = 0.004) and MAE (8.2 vs. 10.5 months; P = 0.002), a higher proportion of bone age predictions within 18 months of chronological age (88.3% vs. 82.2%; P = 0.031) for Institution 1, and a lower MAE (9.5 vs. 11.0 months; P = 0.022) and higher proportion of bone age predictions within 6 months (44.5% vs. 36.4%; P = 0.044) for Institution 2.

CONCLUSION

The Korean model trained using the chronological ages of Korean children and adolescents without known bone development-affecting diseases/conditions as labels performed better in bone age assessment than the GP-based model in the Korean pediatric population. Further validation is required to confirm its accuracy.

Bone age assessment from articular surface and epiphysis using deep neural networks

Bone age assessment is of great significance to genetic diagnosis and endocrine diseases. Traditional bone age diagnosis mainly relies on experienced radiologists to examine the regions of interest in hand radiography, but it is time-consuming and may even lead to a vast error between the diagnosis result and the reference. The existing computer-aided methods predict bone age based on general regions of interest but do not explore specific regions of interest in hand radiography. This paper aims to solve such problems by performing bone age prediction on the articular surface and epiphysis from hand radiography using deep convolutional neural networks. The articular surface and epiphysis datasets are established from the Radiological Society of North America (RSNA) pediatric bone age challenge, where the specific feature regions of the articular surface and epiphysis are manually segmented from hand radiography. Five convolutional neural networks, i.e., ResNet50, SENet, DenseNet-121, EfficientNet-b4, and CSPNet, are employed to improve the accuracy and efficiency of bone age diagnosis in clinical applications. Experiments show that the best-performing model can yield a mean absolute error (MAE) of 7.34 months on the proposed articular surface and epiphysis datasets, which is more accurate and fast than the radiologists. The project is available at https://github.com/YameiDeng/BAANet/, and the annotated dataset is also published at https://doi.org/10.5281/zenodo.7947923.

Prediction of Fishman's skeletal maturity indicators using artificial intelligence

The present study aimed to evaluate the performance of automated skeletal maturation assessment system for Fishman's skeletal maturity indicators (SMI) for the use in dental fields. Skeletal maturity is particularly important in orthodontics for the determination of treatment timing and method. SMI is widely used for this purpose, as it is less time-consuming and practical in clinical use compared to other methods. Thus, the existing automated skeletal age assessment system based on Greulich and Pyle and Tanner-Whitehouse3 methods was further developed to include SMI using artificial intelligence. This hybrid SMI-modified system consists of three major steps: (1) automated detection of region of interest; (2) automated evaluation of skeletal maturity of each region; and (3) SMI stage mapping. The primary validation was carried out using a dataset of 2593 hand-wrist radiographs, and the SMI mapping algorithm was adjusted accordingly. The performance of the final system was evaluated on a test dataset of 711 hand-wrist radiographs from a different institution. The system achieved a prediction accuracy of 0.772 and mean absolute error and root mean square error of 0.27 and 0.604, respectively, indicating a clinically reliable performance. Thus, it can be used to improve clinical efficiency and reproducibility of SMI prediction.

The Cut-Off Value of Serum Anti-Müllerian Hormone Levels for the Diagnosis of Turner Syndrome with Spontaneous Puberty

OBJECTIVE

Preservation of fertility in Turner syndrome (TS) patients may be feasible through cryopreservation of ovarian tissue before follicles begin to disappear. Anti-Müllerian hormone (AMH) is said to be a predictive factor of spontaneous pubertal development in TS. We aimed to determine the cut-off values of AMH for the diagnosis of TS girls with spontaneous puberty. Design and methods: A total of 95 TS patients between 4 and 17 years were evaluated at the Department of Pediatric Genetic Metabolism and Endocrinology from July 2017 to March 2022. Serum AMH, follicle-stimulating hormone (FSH), and luteinizing hormone (LH) levels were analyzed according to age, karyotype, pubertal development, and ultrasound ovarian visualization. Receiver-operating characteristic (ROC) curve analyzes were used to test the utility of AMH for the diagnosis of TS girls with spontaneous puberty.

RESULTS

One-fourth of TS girls aged 8-17 years had spontaneous breast development, with the ratios as follows: 45, X (6/28, 21.4%), mosaicism (7/12, 58.3%), and mosaicism with structural X chromosome abnormalities (SCA) (2/13, 15.4%), SCA (1/13, 7.7%), and Y chromosome (1/3, 33.3%). The AMH cut-off value for the prediction of spontaneous puberty in TS patients was 0.07 ng/ml, with sensitivity and specificity both at 88%. FSH, LH levels, and Karyotypes could not be considered as markers of spontaneous puberty in TS (P > 0.05). A strong relationship was observed between serum AMH levels and spontaneous puberty or ultrasound bilateral ovarian visualization.

CONCLUSIONS

The AMH cut-off value for the prediction of spontaneous puberty in TS girls aged 8-17 years was 0.07 ng/ml, with sensitivity and specificity both at 88%. However, spontaneous puberty in these patients is not predictable based on karyotype or FSH or LH levels.

The Interrater and Intrarater Reliability of the Humeral Head Ossification System and the Proximal Femur Maturity Index Assessments for Patients with Adolescent Idiopathic Scoliosis

Background

Skeletal maturity can evaluate the growth and development potential of children and provide a guide for the management of adolescent idiopathic scoliosis (AIS). Recent studies have demonstrated the advantages of the Humeral Head Ossification System (HHOS) and the Proximal Femur Maturity Index (PFMI), based on standard scoliosis films, in the management of AIS patients. We further assessed the HHOS and the PFMI method's reliability in the interrater and intrarater.

Methods

The data from 38 patients, including the humeral head and proximal femur on standard scoliosis films, were distributed to the eight raters in the form of a PowerPoint presentation. On 38 independent standard spine radiographs, raters utilized the HHOS and PFMI to assign grades. The PPT sequence was randomly changed and then reevaluated 2 weeks later. For every system, the 95% confidence interval (95% CI) and intraclass correlation coefficient (ICC) were calculated to evaluate the interrater and intrarater reliability.

Results

The HHOS was extremely reliable, with an intraobserver ICC of 0.802. In the first round, the interobserver ICC reliability for the HHOS was 0.955 (0.929-0.974), while in the second round, it was 0.939 (0.905-0.964). The PFMI was extremely reliable, with an intraobserver ICC of 0.888. In the first round, the interobserver ICC reliability for the PFMI was 0.967 (0.948-0.981), while in the second round, it was 0.973 (0.957-0.984).

Conclusions

The HHOS and PFMI classifications had excellent reliability. These two methods are beneficial to reduce additional exposure to radiation and expense for AIS. There are advantages and disadvantages to each classification. Clinicians should choose a personalized and reasonable method to assess skeletal maturity, which will assist in the management of adolescent scoliosis patients.

A Study to Evaluate Accuracy and Validity of the EFAI Computer-Aided Bone Age Diagnosis System Compared With Qualified Physicians

Study Objectives

In previous research, we built a deep neural network model based on Inception-Resnet-v2 to predict bone age (EFAI-BAA). The primary objective of the study was to determine if the EFAI-BAA was substantially concordant with the qualified physicians in assessing bone ages. The secondary objective of the study was to determine if the EFAI-BAA was no different in the clinical rating (advanced, normal, or delayed) with the qualified physicians.

Method

This was a retrospective study. The left-hand X-ray images of male subjects aged 3-16 years old and female subjects aged 2-15 years old were collected from China Medical University Hospital (CMUH) and Asia University Hospital (AUH) retrospectively since the trial began until the included image amount reached 368. This was a blinded study. The qualified physicians who ran, read, and interpreted the tests were blinded to the values assessed by the other qualified physicians and the EFAI-BAA.

Results

The concordance correlation coefficient (CCC) between the EFAI-BAA (EFAI-BAA), the evaluation of bone age by physician in Kaohsiung Veterans General Hospital (KVGH), Taichung Veterans General Hospital (TVGH2), and in Taipei Tzu Chi Hospital (TZUCHI-TP) was 0.9828 (95% CI: 0.9790-0.9859, p-value = 0.6782), 0.9739 (95% CI: 0.9681-0.9786, p-value = 0.0202), and 0.9592 (95% CI: 0.9501-0.9666, p-value = 0.4855), respectively.

Conclusion

There was a consistency of bone age assessment between the EFAI-BAA and each one of the three qualified physicians (CCC = 0.9). As the significant difference in the clinical rating was only found between the EFAI-BAA and the qualified physician in TVGH2, the performance of the EFAI-BAA was considered similar to the qualified physicians.

An overview of deep learning applications in precocious puberty and thyroid dysfunction

In the last decade, deep learning methods have garnered a great deal of attention in endocrinology research. In this article, we provide a summary of current deep learning applications in endocrine disorders caused by either precocious onset of adult hormone or abnormal amount of hormone production. To give access to the broader audience, we start with a gentle introduction to deep learning and its most commonly used architectures, and then we focus on the research trends of deep learning applications in thyroid dysfunction classification and precocious puberty diagnosis. We highlight the strengths and weaknesses of various approaches and discuss potential solutions to different challenges. We also go through the practical considerations useful for choosing (and building) the deep learning model, as well as for understanding the thought process behind different decisions made by these models. Finally, we give concluding remarks and future directions.