1
|
Ciritsis A, Rossi C, Eberhard M, Marcon M, Becker AS, Boss A. Automatic classification of ultrasound breast lesions using a deep convolutional neural network mimicking human decision-making. Eur Radiol 2019; 29:5458-5468. [PMID: 30927100 DOI: 10.1007/s00330-019-06118-7] [Citation(s) in RCA: 77] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2018] [Revised: 02/06/2019] [Accepted: 02/15/2019] [Indexed: 12/20/2022]
Abstract
OBJECTIVES To evaluate a deep convolutional neural network (dCNN) for detection, highlighting, and classification of ultrasound (US) breast lesions mimicking human decision-making according to the Breast Imaging Reporting and Data System (BI-RADS). METHODS AND MATERIALS One thousand nineteen breast ultrasound images from 582 patients (age 56.3 ± 11.5 years) were linked to the corresponding radiological report. Lesions were categorized into the following classes: no tissue, normal breast tissue, BI-RADS 2 (cysts, lymph nodes), BI-RADS 3 (non-cystic mass), and BI-RADS 4-5 (suspicious). To test the accuracy of the dCNN, one internal dataset (101 images) and one external test dataset (43 images) were evaluated by the dCNN and two independent readers. Radiological reports, histopathological results, and follow-up examinations served as reference. The performances of the dCNN and the humans were quantified in terms of classification accuracies and receiver operating characteristic (ROC) curves. RESULTS In the internal test dataset, the classification accuracy of the dCNN differentiating BI-RADS 2 from BI-RADS 3-5 lesions was 87.1% (external 93.0%) compared with that of human readers with 79.2 ± 1.9% (external 95.3 ± 2.3%). For the classification of BI-RADS 2-3 versus BI-RADS 4-5, the dCNN reached a classification accuracy of 93.1% (external 95.3%), whereas the classification accuracy of humans yielded 91.6 ± 5.4% (external 94.1 ± 1.2%). The AUC on the internal dataset was 83.8 (external 96.7) for the dCNN and 84.6 ± 2.3 (external 90.9 ± 2.9) for the humans. CONCLUSION dCNNs may be used to mimic human decision-making in the evaluation of single US images of breast lesion according to the BI-RADS catalog. The technique reaches high accuracies and may serve for standardization of highly observer-dependent US assessment. KEY POINTS • Deep convolutional neural networks could be used to classify US breast lesions. • The implemented dCNN with its sliding window approach reaches high accuracies in the classification of US breast lesions. • Deep convolutional neural networks may serve for standardization in US BI-RADS classification.
Collapse
|
Journal Article |
6 |
77 |
2
|
Schawkat K, Ciritsis A, von Ulmenstein S, Honcharova-Biletska H, Jüngst C, Weber A, Gubler C, Mertens J, Reiner CS. Diagnostic accuracy of texture analysis and machine learning for quantification of liver fibrosis in MRI: correlation with MR elastography and histopathology. Eur Radiol 2020; 30:4675-4685. [PMID: 32270315 DOI: 10.1007/s00330-020-06831-8] [Citation(s) in RCA: 38] [Impact Index Per Article: 7.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2020] [Revised: 03/11/2020] [Accepted: 03/24/2020] [Indexed: 12/25/2022]
Abstract
OBJECTIVES To compare the diagnostic accuracy of texture analysis (TA)-derived parameters combined with machine learning (ML) of non-contrast-enhanced T1w and T2w fat-saturated (fs) images with MR elastography (MRE) for liver fibrosis quantification. METHODS In this IRB-approved prospective study, liver MRIs of participants with suspected chronic liver disease who underwent liver biopsy between August 2015 and May 2018 were analyzed. Two readers blinded to clinical and histopathological findings performed TA. The participants were categorized into no or low-stage (0-2) and high-stage (3-4) fibrosis groups. Confusion matrices were calculated using a support vector machine combined with principal component analysis. The diagnostic accuracy of ML-based TA of liver fibrosis and MRE was assessed by area under the receiver operating characteristic curves (AUC). Histopathology served as reference standard. RESULTS A total of 62 consecutive participants (40 men; mean age ± standard deviation, 48 ± 13 years) were included. The accuracy of TA and ML on T1w was 85.7% (95% confidence interval [CI] 63.7-97.0) and 61.9% (95% CI 38.4-81.9) on T2w fs for classification of liver fibrosis into low-stage and high-stage fibrosis. The AUC for TA on T1w was similar to MRE (0.82 [95% CI 0.59-0.95] vs. 0.92 [95% CI 0.71-0.99], p = 0.41), while the AUC for T2w fs was significantly lower compared to MRE (0.57 [95% CI 0.34-0.78] vs. 0.92 [95% CI 0.71-0.99], p = 0.008). CONCLUSION Our results suggest that liver fibrosis can be quantified with TA-derived parameters of T1w when combined with a ML algorithm with similar accuracy compared to MRE. KEY POINTS • Liver fibrosis can be categorized into low-stage fibrosis (0-2) and high-stage fibrosis (3-4) using texture analysis-derived parameters of T1-weighted images with a machine learning approach. • For the differentiation of low-stage fibrosis and high-stage fibrosis, the diagnostic accuracy of texture analysis on T1-weighted images combined with a machine learning algorithm is similar compared to MR elastography.
Collapse
|
Journal Article |
5 |
38 |
3
|
Ciritsis A, Rossi C, Vittoria De Martini I, Eberhard M, Marcon M, Becker AS, Berger N, Boss A. Determination of mammographic breast density using a deep convolutional neural network. Br J Radiol 2018; 92:20180691. [PMID: 30209957 DOI: 10.1259/bjr.20180691] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/21/2023] Open
Abstract
OBJECTIVE High breast density is a risk factor for breast cancer. The aim of this study was to develop a deep convolutional neural network (dCNN) for the automatic classification of breast density based on the mammographic appearance of the tissue according to the American College of Radiology Breast Imaging Reporting and Data System (ACR BI-RADS) Atlas. METHODS In this study, 20,578 mammography single views from 5221 different patients (58.3 ± 11.5 years) were downloaded from the picture archiving and communications system of our institution and automatically sorted according to the ACR density (a-d) provided by the corresponding radiological reports. A dCNN with 11 convolutional layers and 3 fully connected layers was trained and validated on an augmented dataset. The model was finally tested on two different datasets against: i) the radiological reports and ii) the consensus decision of two human readers. None of the test datasets was part of the dataset used for the training and validation of the algorithm. RESULTS The optimal number of epochs was 91 for medio-lateral oblique (MLO) projections and 94 for cranio-caudal projections (CC), respectively. Accuracy for MLO projections obtained on the validation dataset was 90.9% (CC: 90.1%). Tested on the first test dataset of mammographies (850 MLO and 880 CC), the algorithm showed an accordance with the corresponding radiological reports of 71.7% for MLO and of 71.0% for CC. The agreement with the radiological reports improved in the differentiation between dense and fatty breast for both projections (MLO = 88.6% and CC = 89.9%). In the second test dataset of 200 mammographies, a good accordance was found between the consensus decision of the two readers on both, the MLO-model (92.2%) and the right craniocaudal-model (87.4%). In the differentiation between fatty (ACR A/B) and dense breasts (ACR C/D), the agreement reached 99% for the MLO and 96% for the CC projections, respectively. CONCLUSIONS The dCNN allows for accurate classification of breast density based on the ACR BI-RADS system. The proposed technique may allow accurate, standardized, and observer independent breast density evaluation of mammographies. ADVANCES IN KNOWLEDGE Standardized classification of mammographies by a dCNN could lead to a reduction of falsely classified breast densities, thereby allowing for a more accurate breast cancer risk assessment for the individual patient and a more reliable decision, whether additional ultrasound is recommended.
Collapse
|
Journal Article |
7 |
18 |
4
|
Borkowski K, Rossi C, Ciritsis A, Marcon M, Hejduk P, Stieb S, Boss A, Berger N. Fully automatic classification of breast MRI background parenchymal enhancement using a transfer learning approach. Medicine (Baltimore) 2020; 99:e21243. [PMID: 32702902 PMCID: PMC7373599 DOI: 10.1097/md.0000000000021243] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
Marked enhancement of the fibroglandular tissue on contrast-enhanced breast magnetic resonance imaging (MRI) may affect lesion detection and classification and is suggested to be associated with higher risk of developing breast cancer. The background parenchymal enhancement (BPE) is qualitatively classified according to the BI-RADS atlas into the categories "minimal," "mild," "moderate," and "marked." The purpose of this study was to train a deep convolutional neural network (dCNN) for standardized and automatic classification of BPE categories.This IRB-approved retrospective study included 11,769 single MR images from 149 patients. The MR images were derived from the subtraction between the first post-contrast volume and the native T1-weighted images. A hierarchic approach was implemented relying on 2 dCNN models for detection of MR-slices imaging breast tissue and for BPE classification, respectively. Data annotation was performed by 2 board-certified radiologists. The consensus of the 2 radiologists was chosen as reference for BPE classification. The clinical performances of the single readers and of the dCNN were statistically compared using the quadratic Cohen's kappa.Slices depicting the breast were classified with training, validation, and real-world (test) accuracies of 98%, 96%, and 97%, respectively. Over the 4 classes, the BPE classification was reached with mean accuracies of 74% for training, 75% for the validation, and 75% for the real word dataset. As compared to the reference, the inter-reader reliabilities for the radiologists were 0.780 (reader 1) and 0.679 (reader 2). On the other hand, the reliability for the dCNN model was 0.815.Automatic classification of BPE can be performed with high accuracy and support the standardization of tissue classification in MRI.
Collapse
|
research-article |
5 |
16 |
5
|
Ciritsis A, Rossi C, Marcon M, Van VDP, Boss A. Accelerated diffusion-weighted imaging for lymph node assessment in the pelvis applying simultaneous multislice acquisition: A healthy volunteer study. Medicine (Baltimore) 2018; 97:e11745. [PMID: 30095628 PMCID: PMC6133413 DOI: 10.1097/md.0000000000011745] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/06/2023] Open
Abstract
To evaluate the feasibility of accelerated simultaneous multislice diffusion weighted sequences (SMS-DWI) for lymph node detection in the abdominopelvic region. Sequences were evaluated regarding the number and depiction of lymph nodes detected with SMS-DWI compared with conventional diffusion weighted sequences, the most suitable SMS- acceleration factor, signal-to-noise ratio (SNR), and the overall acquisition time (TA).Eight healthy volunteers (4 men, 4 women; age range 21-39 years; median age 25 years) were examined in the pelvic region at 3T using a conventional DWI sequence and a SMS DWI sequence with different acceleration factors (AF: 2-3). Moreover, a SMS DWI sequence with AF 3 and higher slice resolution was applied. For morphological correlation of the lymph nodes and as a reference standard, an isotropic 3-dimensional T2-weighted fast-spin-echo sequence with high sampling efficiency (SPACE) was acquired. Two radiologists reviewed each DWI sequence and assessed the number of lymph nodes and the overall image quality. For each DWI sequence, SNR, SNR efficiency per time, contrast to noise (CNR), and ADC values were calculated. Values were statistically compared using a Wilcoxon test (P < .05).Overall, scan time of SMS-DWI with AF2 (AF3) decreased by 46.9% (57.2%) with respect to the conventional DWI. Compared with the SPACE sequence, the detection rate was 89.6% for conventional DWI, 69.4% for SMS-DWI with AF2, and 59.9% for SMS-DWI with AF3. The highly resolved SMS-DWI with AF3 leads to a scan time reduction of 46.9% and detection rate of 83.0%. SNR and CNR were lower in the accelerated sequences (up to 51.0%, P < .001) as compared with the conventional DWI. SNR efficiency decreased to 19.3% for AF2 and to 31.3% for AF3. In the highly resolved dataset, an SNR efficiency reduction of 51.2% was found.This study showed that lymph node detection in the abdominopelvic region with accelerated SMS-DWI sequences is feasible whereby an AF of 2 represents the best compromise between image quality, SNR, CNR, TA, and detection rate.
Collapse
|
Evaluation Study |
7 |
13 |
6
|
Hejduk P, Marcon M, Unkelbach J, Ciritsis A, Rossi C, Borkowski K, Boss A. Fully automatic classification of automated breast ultrasound (ABUS) imaging according to BI-RADS using a deep convolutional neural network. Eur Radiol 2022; 32:4868-4878. [PMID: 35147776 PMCID: PMC9213284 DOI: 10.1007/s00330-022-08558-0] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2021] [Revised: 12/14/2021] [Accepted: 12/26/2021] [Indexed: 12/15/2022]
Abstract
PURPOSE The aim of this study was to develop and test a post-processing technique for detection and classification of lesions according to the BI-RADS atlas in automated breast ultrasound (ABUS) based on deep convolutional neural networks (dCNNs). METHODS AND MATERIALS In this retrospective study, 645 ABUS datasets from 113 patients were included; 55 patients had lesions classified as high malignancy probability. Lesions were categorized in BI-RADS 2 (no suspicion of malignancy), BI-RADS 3 (probability of malignancy < 3%), and BI-RADS 4/5 (probability of malignancy > 3%). A deep convolutional neural network was trained after data augmentation with images of lesions and normal breast tissue, and a sliding-window approach for lesion detection was implemented. The algorithm was applied to a test dataset containing 128 images and performance was compared with readings of 2 experienced radiologists. RESULTS Results of calculations performed on single images showed accuracy of 79.7% and AUC of 0.91 [95% CI: 0.85-0.96] in categorization according to BI-RADS. Moderate agreement between dCNN and ground truth has been achieved (κ: 0.57 [95% CI: 0.50-0.64]) what is comparable with human readers. Analysis of whole dataset improved categorization accuracy to 90.9% and AUC of 0.91 [95% CI: 0.77-1.00], while achieving almost perfect agreement with ground truth (κ: 0.82 [95% CI: 0.69-0.95]), performing on par with human readers. Furthermore, the object localization technique allowed the detection of lesion position slice-wise. CONCLUSIONS Our results show that a dCNN can be trained to detect and distinguish lesions in ABUS according to the BI-RADS classification with similar accuracy as experienced radiologists. KEY POINTS • A deep convolutional neural network (dCNN) was trained for classification of ABUS lesions according to the BI-RADS atlas. • A sliding-window approach allows accurate automatic detection and classification of lesions in ABUS examinations.
Collapse
|
research-article |
3 |
13 |
7
|
Ciritsis A, Boss A, Rossi C. Automated pixel-wise brain tissue segmentation of diffusion-weighted images via machine learning. NMR IN BIOMEDICINE 2018; 31:e3931. [PMID: 29697165 DOI: 10.1002/nbm.3931] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/27/2017] [Revised: 02/27/2018] [Accepted: 03/09/2018] [Indexed: 06/08/2023]
Abstract
The diffusion-weighted (DW) MR signal sampled over a wide range of b-values potentially allows for tissue differentiation in terms of cellularity, microstructure, perfusion, and T2 relaxivity. This study aimed to implement a machine learning algorithm for automatic brain tissue segmentation from DW-MRI datasets, and to determine the optimal sub-set of features for accurate segmentation. DWI was performed at 3 T in eight healthy volunteers using 15 b-values and 20 diffusion-encoding directions. The pixel-wise signal attenuation, as well as the trace and fractional anisotropy (FA) of the diffusion tensor, were used as features to train a support vector machine classifier for gray matter, white matter, and cerebrospinal fluid classes. The datasets of two volunteers were used for validation. For each subject, tissue classification was also performed on 3D T1 -weighted data sets with a probabilistic framework. Confusion matrices were generated for quantitative assessment of image classification accuracy in comparison with the reference method. DWI-based tissue segmentation resulted in an accuracy of 82.1% on the validation dataset and of 82.2% on the training dataset, excluding relevant model over-fitting. A mean Dice coefficient (DSC) of 0.79 ± 0.08 was found. About 50% of the classification performance was attributable to five features (i.e. signal measured at b-values of 5/10/500/1200 s/mm2 and the FA). This reduced set of features led to almost identical performances for the validation (82.2%) and the training (81.4%) datasets (DSC = 0.79 ± 0.08). Machine learning techniques applied to DWI data allow for accurate brain tissue segmentation based on both morphological and functional information.
Collapse
|
|
7 |
12 |
8
|
Marcon M, Ciritsis A, Rossi C, Becker AS, Berger N, Wurnig MC, Wagner MW, Frauenfelder T, Boss A. Diagnostic performance of machine learning applied to texture analysis-derived features for breast lesion characterisation at automated breast ultrasound: a pilot study. Eur Radiol Exp 2019; 3:44. [PMID: 31676937 PMCID: PMC6825080 DOI: 10.1186/s41747-019-0121-6] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2019] [Accepted: 08/28/2019] [Indexed: 12/31/2022] Open
Abstract
Background Our aims were to determine if features derived from texture analysis (TA) can distinguish normal, benign, and malignant tissue on automated breast ultrasound (ABUS); to evaluate whether machine learning (ML) applied to TA can categorise ABUS findings; and to compare ML to the analysis of single texture features for lesion classification. Methods This ethically approved retrospective pilot study included 54 women with benign (n = 38) and malignant (n = 32) solid breast lesions who underwent ABUS. After manual region of interest placement along the lesions’ margin as well as the surrounding fat and glandular breast tissue, 47 texture features (TFs) were calculated for each category. Statistical analysis (ANOVA) and a support vector machine (SVM) algorithm were applied to the texture feature to evaluate the accuracy in distinguishing (i) lesions versus normal tissue and (ii) benign versus malignant lesions. Results Skewness and kurtosis were the only TF significantly different among all the four categories (p < 0.000001). In subsets (i) and (ii), a maximum area under the curve of 0.86 (95% confidence interval [CI] 0.82–0.88) for energy and 0.86 (95% CI 0.82–0.89) for entropy were obtained. Using the SVM algorithm, a maximum area under the curve of 0.98 for both subsets was obtained with a maximum accuracy of 94.4% in subset (i) and 90.7% in subset (ii). Conclusions TA in combination with ML might represent a useful diagnostic tool in the evaluation of breast imaging findings in ABUS. Applying ML techniques to TFs might be superior compared to the analysis of single TF. Electronic supplementary material The online version of this article (10.1186/s41747-019-0121-6) contains supplementary material, which is available to authorized users.
Collapse
|
Research Support, Non-U.S. Gov't |
6 |
10 |
9
|
Ho MJ, Ciritsis A, Manoliu A, Stieltjes B, Marcon M, Andreisek G, Kuhn FP. Diffusion Tensor Imaging of the Brachial Plexus: A Comparison between Readout-segmented and Conventional Single-shot Echo-planar Imaging. Magn Reson Med Sci 2018; 18:150-157. [PMID: 30416178 PMCID: PMC6460122 DOI: 10.2463/mrms.mp.2018-0004] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
Purpose: Diffusion tensor imaging (DTI) adds functional information to morphological magnetic resonance neurography (MRN) in the assessment of the brachial nerve plexus. To determine the most appropriate pulse sequence in scan times suited for diagnostic imaging in clinical routine, we compared image quality between simultaneous multi-slice readout-segmented (rs-DTI) and conventional single-shot (ss-DTI) echo-planar imaging techniques. Methods: Institutional Review Board (IRB) approved study including 10 healthy volunteers. The supraclavicular brachial plexus, covering the nerve roots and trunks from C5 to C7, was imaged on both sides with rs-DTI and ss-DTI. Both sequences were acquired in scan times <7 min with b-values of 900 s/mm2 and with isotropic spatial resolution. Results: In rs-DTI image, the overall quality was significantly better and distortion artifacts were significantly lower (P = 0.001–0.002 and P = 0.001–0.002, respectively) for both readers. In ss-DTI, a trend toward lower degree of ghosting and motion artifacts was elicited (reader 1, P = 0.121; reader 2, P = 0.264). No significant differences between the two DTI techniques were found for signal-to-noise ratios (SNR), contrast-to-noise ratios (CNR) and fractional anisotropy (FA) (P ≥ 0.475, P ≥ 0.624, and P ≥ 0.169, respectively). Interreader agreement for all examined parameters and all sequences ranged from intraclass correlation coefficient (ICC) 0.064 to 0.905 and Kappa 0.40 to 0.851. Conclusion: Incomparable acquisition times rs-DTI showed higher image quality and less distortion artifacts than ss-DTI. The trend toward a higher degree of ghosting and motion artifacts in rs-DTI did not deteriorate image quality to a significant degree. Thus, rs-DTI should be considered for functional MRN of the brachial plexus.
Collapse
|
Journal Article |
7 |
9 |
10
|
Schönenberger C, Hejduk P, Ciritsis A, Marcon M, Rossi C, Boss A. Classification of Mammographic Breast Microcalcifications Using a Deep Convolutional Neural Network: A BI-RADS-Based Approach. Invest Radiol 2021; 56:224-231. [PMID: 33038095 DOI: 10.1097/rli.0000000000000729] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
MATERIALS AND METHODS Over 56,000 images of 268 mammograms from 94 patients were labeled to 3 classes according to the BI-RADS standard: "no microcalcifications" (BI-RADS 1), "probably benign microcalcifications" (BI-RADS 2/3), and "suspicious microcalcifications" (BI-RADS 4/5). Using the preprocessed images, a dCNN was trained and validated, generating 3 types of models: BI-RADS 4 cohort, BI-RADS 5 cohort, and BI-RADS 4 + 5 cohort. For the final validation of the trained dCNN models, a test data set consisting of 141 images of 51 mammograms from 26 patients labeled according to the corresponding BI-RADS classification from the radiological reports was applied. The performances of the dCNN models were evaluated, classifying each of the mammograms and computing the accuracy in comparison to the classification from the radiological reports. For visualization, probability maps of the classification were generated. RESULTS The accuracy on the validation set after 130 epochs was 99.5% for the BI-RADS 4 cohort, 99.6% for the BI-RADS 5 cohort, and 98.1% for the BI-RADS 4 + 5 cohort. Confusion matrices of the "real-world" test data set for the 3 cohorts were generated where the radiological reports served as ground truth. The resulting accuracy was 39.0% for the BI-RADS 4 cohort, 80.9% for BI-RADS 5 cohort, and 76.6% for BI-RADS 4 + 5 cohort. The probability maps exhibited excellent image quality with correct classification of microcalcification distribution. CONCLUSIONS The dCNNs can be trained to successfully classify microcalcifications on mammograms according to the BI-RADS classification system in order to act as a standardized quality control tool providing the expertise of a team of radiologists.
Collapse
|
|
4 |
8 |
11
|
Ciritsis A, Horbach A, Staat M, Kuhl CK, Kraemer NA. Porosity and tissue integration of elastic mesh implants evaluatedin vitroandin vivo. J Biomed Mater Res B Appl Biomater 2017; 106:827-833. [DOI: 10.1002/jbm.b.33877] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/06/2016] [Revised: 01/03/2017] [Accepted: 02/20/2017] [Indexed: 12/26/2022]
|
|
8 |
8 |
12
|
Otto J, Kuehnert N, Kraemer NA, Ciritsis A, Hansen NL, Kuhl C, Busch D, Peter Neumann U, Klinge U, Conze KJ. First in vivo visualization of MRI-visible IPOM in a rabbit model. J Biomed Mater Res B Appl Biomater 2014; 102:1165-9. [DOI: 10.1002/jbm.b.33098] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2013] [Revised: 11/10/2013] [Accepted: 12/17/2013] [Indexed: 11/07/2022]
|
|
11 |
8 |
13
|
Phi van V, Reiner CS, Klarhoefer M, Ciritsis A, Eberhardt C, Wurnig MC, Rossi C. Diffusion tensor imaging of the abdominal organs: Influence of oriented intravoxel flow compartments. NMR IN BIOMEDICINE 2019; 32:e4159. [PMID: 31397037 DOI: 10.1002/nbm.4159] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/09/2018] [Revised: 06/24/2019] [Accepted: 06/26/2019] [Indexed: 06/10/2023]
Abstract
Water flow in partially oriented intravoxel compartments mimics an anisotropic fast-diffusion regime, which contributes to the signal attenuation in diffusion-weighted images. In the abdominal organs, this flow may reflect physiological fluid movements (eg, tubular urine flow in kidneys, or bile flow through the liver) and have a clinical relevance. This study investigated the influence of anisotropic intravoxel water flow on diffusion tensor imaging (DTI) of the abdominal organs. Diffusion-weighted images were acquired in five healthy volunteers using an EPI sequence with diffusion preparation (TR/TE: 1000 ms/71 ms; b-values: 0, 10, 20, 40, 70, 120, 250, 450, 700, 1000 s/mm2 ; 12 noncollinear diffusion-encoding directions). DTI of liver and kidneys was performed assuming (i) monoexponential decay of the diffusion-weighted signal, and (ii) accounting for potential anisotropy of the fast-diffusion compartments using a tensorial generalization of the IVIM model. Additionally, potential dependency of the metrics of the tensors from the anatomical location was evaluated. Significant differences in the metrics of the diffusion tensor (DT) were found in both liver and kidneys when comparing the two models. In both organs, the trace and the fractional anisotropy of the DT were significantly higher in the monoexponential model than when accounting for perfusion. The comparison of areas of the liver proximal to the hilum with distal regions and of renal cortex with the medulla also proved a location dependency of the size of the fast-diffusion compartments. Pseudo-diffusion correction in DTI enables the assessment of the solid parenchyma regardless of the organ perfusion or other pseudo-diffusive fluid movements. This may have a clinical relevance in the assessment of parenchymal pathologies (eg, liver fibrosis). The fast pseudo-diffusion components present a detectable anisotropy, which may reflect the hepatic microcirculation or other sources of mesoscopic fluid movement in the abdominal organs.
Collapse
|
|
6 |
6 |
14
|
Ciritsis A, Truhn D, Hansen NL, Otto J, Kuhl CK, Kraemer NA. Positive Contrast MRI Techniques for Visualization of Iron-Loaded Hernia Mesh Implants in Patients. PLoS One 2016; 11:e0155717. [PMID: 27192201 PMCID: PMC4871409 DOI: 10.1371/journal.pone.0155717] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2015] [Accepted: 05/03/2016] [Indexed: 01/07/2023] Open
Abstract
Object In MRI, implants and devices can be delineated via susceptibility artefacts. To discriminate susceptibility voids from proton-free structures, different positive contrast techniques were implemented. The purpose of this study was to evaluate a pulse sequence-based positive contrast technique (PCSI) and a post-processing susceptibility gradient mapping algorithm (SGM) for visualization of iron loaded mesh implants in patients. Material and Methods Five patients with iron-loaded MR-visible inguinal hernia mesh implants were examined at 1.5 Tesla. A gradient echo sequence (GRE; parameters: TR: 8.3ms; TE: 4.3ms; NSA:2; FA:20°; FOV:350mm²) and a PCSI sequence (parameters: TR: 25ms; TE: 4.6ms; NSA:4; FA:20°; FOV:350mm²) with on-resonant proton suppression were performed. SGM maps were calculated using two algorithms. Image quality and mesh delineation were independently evaluated by three radiologists. Results On GRE, the iron-loaded meshes generated distinct susceptibility-induced signal voids. PCSI exhibited susceptibility differences including the meshes as hyperintense signals. SGM exhibited susceptibility differences with positive contrast. Visually, the different algorithms presented no significant differences. Overall, the diagnostic value was rated best in GRE whereas PCSI and SGM were barely “sufficient”. Conclusion Both “positive contrast” techniques depicted implanted meshes with hyperintense signal. SGM comes without additional acquisition time and can therefore be utilized in every patient.
Collapse
|
|
9 |
6 |
15
|
Lambertz A, van den Hil LCL, Ciritsis A, Eickhoff R, Kraemer NA, Bouvy ND, Müllen A, Klinge U, Neumann UP, Klink CD. MRI Evaluation of an Elastic TPU Mesh under Pneumoperitoneum in IPOM Position in a Porcine Model. J INVEST SURG 2017; 31:185-191. [PMID: 28594257 DOI: 10.1080/08941939.2017.1301599] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
BACKGROUND The frequency of laparoscopic approaches increased in hernia surgery over the past years. After mesh placement in IPOM position, the real extent of the meshes configurational changes after termination of pneumoperitoneum is still largely unknown. To prevent a later mesh folding it might be useful to place the mesh while it is kept under tension. Conventionally used meshes may lose their Effective Porosity under these conditions due to poor elastic properties. The aim of this study was to evaluate a newly developed elastic thermoplastic polyurethane (TPU) containing mesh that retains its Effective Porosity under mechanical strain in IPOM position in a porcine model. It was visualized under pneumoperitoneum using MRI in comparison to polyvinylidenefluoride (PVDF) meshes with similar structure. METHODS In each of ten minipigs, a mesh (TPU containing or native PVDF, 10 × 20 cm) was randomly placed in IPOM position at the center of the abdominal wall. After 8 weeks, six pigs underwent MRI evaluation with and without pneumoperitoneum to assess the visibility and elasticity of the mesh. Finally, pigs were euthanized and abdominal walls were explanted for histological and immunohistochemical assessment. The degree of adhesion formation was documented. RESULTS Laparoscopic implantation of elastic TPU meshes in IPOM position was feasible and safe in a minipig model. Mesh position could be precisely visualized and assessed with and without pneumoperitoneum using MRI after 8 weeks. Elastic TPU meshes showed a significantly higher surface increase under pneumoperitoneum in comparison to PVDF. Immunohistochemically, the amount of CD45-positive cells was significantly lower and the Collagen I/III ratio was significantly higher in TPU meshes after 8 weeks. There were no differences regarding adhesion formation between study groups. CONCLUSIONS The TPU mesh preserves its elastic properties in IPOM position in a porcine model after 8 weeks. Immunohistochemistry indicates superior biocompatibility regarding CD45-positive cells and Collagen I/III ratio in comparison to PVDF meshes with a similar structure.
Collapse
|
Journal Article |
8 |
5 |
16
|
Landsmann A, Wieler J, Hejduk P, Ciritsis A, Borkowski K, Rossi C, Boss A. Applied Machine Learning in Spiral Breast-CT: Can We Train a Deep Convolutional Neural Network for Automatic, Standardized and Observer Independent Classification of Breast Density? Diagnostics (Basel) 2022; 12:diagnostics12010181. [PMID: 35054348 PMCID: PMC8775263 DOI: 10.3390/diagnostics12010181] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Revised: 01/05/2022] [Accepted: 01/11/2022] [Indexed: 02/05/2023] Open
Abstract
The aim of this study was to investigate the potential of a machine learning algorithm to accurately classify parenchymal density in spiral breast-CT (BCT), using a deep convolutional neural network (dCNN). In this retrospectively designed study, 634 examinations of 317 patients were included. After image selection and preparation, 5589 images from 634 different BCT examinations were sorted by a four-level density scale, ranging from A to D, using ACR BI-RADS-like criteria. Subsequently four different dCNN models (differences in optimizer and spatial resolution) were trained (70% of data), validated (20%) and tested on a “real-world” dataset (10%). Moreover, dCNN accuracy was compared to a human readout. The overall performance of the model with lowest resolution of input data was highest, reaching an accuracy on the “real-world” dataset of 85.8%. The intra-class correlation of the dCNN and the two readers was almost perfect (0.92) and kappa values between both readers and the dCNN were substantial (0.71–0.76). Moreover, the diagnostic performance between the readers and the dCNN showed very good correspondence with an AUC of 0.89. Artificial Intelligence in the form of a dCNN can be used for standardized, observer-independent and reliable classification of parenchymal density in a BCT examination.
Collapse
|
|
3 |
3 |
17
|
Gomolka RS, Ciritsis A, Meier A, Rossi C. Quantification of sodium T1 in abdominal tissues at 3 T. MAGNETIC RESONANCE MATERIALS IN PHYSICS BIOLOGY AND MEDICINE 2019; 33:439-446. [DOI: 10.1007/s10334-019-00786-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/14/2019] [Revised: 09/10/2019] [Accepted: 10/04/2019] [Indexed: 02/02/2023]
|
|
6 |
3 |
18
|
Choschzick M, Alyahiaoui M, Ciritsis A, Rossi C, Gut A, Hejduk P, Boss A. Deep learning for the standardized classification of Ki-67 in vulva carcinoma: A feasibility study. Heliyon 2021; 7:e07577. [PMID: 34386617 PMCID: PMC8346648 DOI: 10.1016/j.heliyon.2021.e07577] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Revised: 02/02/2021] [Accepted: 07/12/2021] [Indexed: 12/12/2022] Open
Abstract
Background The aim of this study is to demonstrate the feasibility of automatic classification of Ki-67 histological immunostainings in patients with squamous cell carcinoma of the vulva using a deep convolutional neural network (dCNN). Material and methods For evaluation of the dCNN, we used 55 well characterized squamous cell carcinomas of the vulva in a tissue microarray (TMA) format in this retrospective study. The tumor specimens were classified in 3 different categories C1 (0-2%), C2 (2-20%) and C3 (>20%), representing the relation of the number of KI-67 positive tumor cells to all cancer cells on the TMA spot. Representative areas of the spots were manually labeled by extracting images of 351 × 280 pixels. A dCNN with 13 convolutional layers was used for the evaluation. Two independent pathologists classified 45 labeled images in order to compare the dCNN's results to human readouts. Results Using a small labeled dataset with 1020 images with equal distribution among classes, the dCNN reached an accuracy of 90.9% (93%) for the training (validation) data. Applying a larger dataset with additional 1017 labeled images resulted in an accuracy of 96.1% (91.4%) for the training (validation) dataset. For the human readout, there were no significant differences between the pathologists and the dCNN in Ki-67 classification results. Conclusion The dCNN is capable of a standardized classification of Ki-67 staining in vulva carcinoma; therefore, it may be suitable for quality control and standardization in the assessment of tumor grading.
Collapse
|
Journal Article |
4 |
3 |
19
|
Gomolka RS, Ciritsis A, Rossi C. 23 Na-T 1 quantification with saturation recovery TrueFISP and variable flip angle GRE at 3T: A phantom study. Magn Reson Med 2020; 84:3300-3307. [PMID: 32544302 DOI: 10.1002/mrm.28333] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2019] [Revised: 04/11/2020] [Accepted: 05/01/2020] [Indexed: 11/05/2022]
Abstract
PURPOSE The aim of the current study was to compare the reproducibility of sodium (23 Na)-T1 estimation using a centric-reordered saturation recovery (SR) true fast imaging with steady-state precession (TrueFISP) and a variable flip angle (VFA) spoiled gradient echo (GRE). Additionally, we evaluated the effect of spatial averaging on 23 Na-T1 estimation by the two methods. METHODS Measurements were performed in the phantom, consisting of 10 dm3 volume rectangular polyethylene container filled with distilled water solution of 0.6% NaCl + 0.004% CuSO4 , using a dual-tunable 23 Na/1 H coil at 3 Tesla. 23 Na images were acquired for FOV = 384 × 384 mm2 and voxel size = 6 × 6 × 6 mm3 using: (1) TrueFISP: TR/TE = 900/1.5 ms, flip angle = 90°, bandwidth = 450 Hz/px, and (2) GRE: TR/TE = 30/1.5 ms, bandwidth = 350 Hz/px. 23 Na-T1 weightings were obtained with nonselective saturation prepulses delayed from the center of the k-space acquisition by 25/40/60/130/280 ms (SR-TrueFISP) and by applying different nominal flip angles: 10°/30°/50°/70°/90° (VFA-GRE). Both sequences were acquired twice, applying 20 and 30 spatial averages. The resulting images were B1 -corrected with a double-angle GRE method. RESULTS Image acquisition varied from 5:41 to 9:37 for TrueFISP and from 12:48 to 19:12 min for GRE using 20 and 30 spatial averages, respectively. Higher averaging increased the acquisition time by 53% and mean SNR at scan < 10%, without an effect on 23 Na-T1 estimations with both methods (SR-Truefisp |Δ| = 1.58 ms, VFA-GRE |Δ| = 0.53 ms; for SNR P < .001). Overall, mean ± SD of 23 Na-T1 was found as 51 ± 3 ms with SR-TrueFISP and 53 ± 2 ms with VFA-GRE. CONCLUSION Both SR-TrueFISP and VFA-GRE provided similar 23 Na-T1 estimates based on the phantom measurements with isotropic resolution.
Collapse
|
|
5 |
1 |
20
|
Hansen NL, Otto J, Ciritsis A, Barabasch A, Klinge U, Kuhl C, Krämer N. MRT-Evaluation des Therapieerfolgs von Leistenhernien in Patienten mit eisenhaltigen Netzimplantaten. ROFO-FORTSCHR RONTG 2014. [DOI: 10.1055/s-0034-1372827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
|
11 |
|
21
|
Stieb S, Klarhoefer M, Finkenstaedt T, Wurnig MC, Becker AS, Ciritsis A, Rossi C. Correction for fast pseudo-diffusive fluid motion contaminations in diffusion tensor imaging. Magn Reson Imaging 2019; 66:50-56. [PMID: 31655141 DOI: 10.1016/j.mri.2019.09.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Revised: 07/18/2019] [Accepted: 09/15/2019] [Indexed: 11/26/2022]
Abstract
In this prospective study, we quantified the fast pseudo-diffusion contamination by blood perfusion or cerebrospinal fluid (CSF) intravoxel incoherent movements on the measurement of the diffusion tensor metrics in healthy brain tissue. Diffusion-weighted imaging (TR/TE = 4100 ms/90 ms; b-values: 0, 5, 10, 20, 35, 55, 80, 110, 150, 200, 300, 500, 750, 1000, 1300 s/mm2, 20 diffusion-encoding directions) was performed on a cohort of five healthy volunteers at 3 Tesla. The projections of the diffusion tensor along each diffusion-encoding direction were computed using a two b-value approach (2b), by fitting the signal to a monoexponential curve (mono), and by correcting for fast pseudo-diffusion compartments using the biexponential intravoxel incoherent motion model (IVIM) (bi). Fractional anisotropy (FA) and mean diffusivity (MD) of the diffusion tensor were quantified in regions of interest drawn over white matter areas, gray matter areas, and the ventricles. A significant dependence of the MD from the evaluation method was found in all selected regions. A lower MD was computed when accounting for the fast-diffusion compartments. A larger dependence was found in the nucleus caudatus (bi: median 0.86 10-3 mm2/s, Δ2b: -11.2%, Δmono: -14.4%; p = 0.007), in the anterior horn (bi: median 2.04 10-3 mm2/s, Δ2b: -9.4%, Δmono: -11.5%, p = 0.007) and in the posterior horn of the lateral ventricles (bi: median 2.47 10-3 mm2/s, Δ2b: -5.5%, Δmono: -11.7%; p = 0.007). Also for the FA, the signal modeling affected the computation of the anisotropy metrics. The deviation depended on the evaluated region with significant differences mainly in the nucleus caudatus (bi: median 0.15, Δ2b: +39.3%, Δmono: +14.7%; p = 0.022) and putamen (bi: median 0.19, Δ2b: +3.1%, Δmono: +17.3%; p = 0.015). Fast pseudo-diffusive regimes locally affect diffusion tensor imaging (DTI) metrics in the brain. Here, we propose the use of an IVIM-based method for correction of signal contaminations through CSF or perfusion.
Collapse
|
|
6 |
|
22
|
Nowakowska S, Vescoli V, Schnitzler T, Ruppert C, Borkowski K, Boss A, Rossi C, Wein B, Ciritsis A. Technical feasibility of automated blur detection in digital mammography using convolutional neural network. Eur Radiol Exp 2024; 8:129. [PMID: 39556167 PMCID: PMC11574226 DOI: 10.1186/s41747-024-00527-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Accepted: 10/17/2024] [Indexed: 11/19/2024] Open
Abstract
BACKGROUND The presence of a blurred area, depending on its localization, in a mammogram can limit diagnostic accuracy. The goal of this study was to develop a model for automatic detection of blur in diagnostically relevant locations in digital mammography. METHODS A retrospective dataset consisting of 152 examinations acquired with mammography machines from three different vendors was utilized. The blurred areas were contoured by expert breast radiologists. Normalized Wiener spectra (nWS) were extracted in a sliding window manner from each mammogram. These spectra served as input for a convolutional neural network (CNN) generating the probability of the spectra originating from a blurred region. The resulting blur probability mask, upon thresholding, facilitated the classification of a mammogram as either blurred or sharp. Ground truth for the test set was defined by the consensus of two radiologists. RESULTS A significant correlation between the view (p < 0.001), as well as between the laterality and the presence of blur (p = 0.004) was identified. The developed model AUROC of 0.808 (95% confidence interval 0.794-0.821) aligned with the consensus in 78% (67-83%) of mammograms classified as blurred. For mammograms classified by consensus as sharp, the model achieved agreement in 75% (67-83%) of them. CONCLUSION A model for blur detection was developed and assessed. The results indicate that a robust approach to blur detection, based on feature extraction in frequency space, tailored to radiologist expertise regarding clinical relevance, could eliminate the subjectivity associated with the visual assessment. RELEVANCE STATEMENT This blur detection model, if implemented in clinical practice, could provide instantaneous feedback to technicians, allowing for prompt mammogram retakes and ensuring that only high-quality mammograms are sent for screening and diagnostic tasks. KEY POINTS Blurring in mammography limits radiologist interpretation and diagnostic accuracy. This objective blur detection tool ensures image quality, and reduces retakes and unnecessary exposures. Wiener spectrum analysis and CNN enabled automated blur detection in mammography.
Collapse
|
|
1 |
|
23
|
Hansen NL, Barabasch A, Distelmaier M, Ciritsis A, Kühnert N, Otto J, Conze J, Kuhl CK, Krämer NA. Erstmalige MRT-Visualisierung von chirurgischen eisenhaltigen Netzimplantaten bei Patienten mit Leistenhernie. ROFO-FORTSCHR RONTG 2013. [DOI: 10.1055/s-0033-1346392] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
|
12 |
|
24
|
Nowakowska S, Borkowski K, Ruppert C, Hejduk P, Ciritsis A, Landsmann A, Marcon M, Berger N, Boss A, Rossi C. Explainable Precision Medicine in Breast MRI: A Combined Radiomics and Deep Learning Approach for the Classification of Contrast Agent Uptake. Bioengineering (Basel) 2024; 11:556. [PMID: 38927793 PMCID: PMC11200390 DOI: 10.3390/bioengineering11060556] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Revised: 05/24/2024] [Accepted: 05/29/2024] [Indexed: 06/28/2024] Open
Abstract
In DCE-MRI, the degree of contrast uptake in normal fibroglandular tissue, i.e., background parenchymal enhancement (BPE), is a crucial biomarker linked to breast cancer risk and treatment outcome. In accordance with the Breast Imaging Reporting & Data System (BI-RADS), it should be visually classified into four classes. The susceptibility of such an assessment to inter-reader variability highlights the urgent need for a standardized classification algorithm. In this retrospective study, the first post-contrast subtraction images for 27 healthy female subjects were included. The BPE was classified slice-wise by two expert radiologists. The extraction of radiomic features from segmented BPE was followed by dataset splitting and dimensionality reduction. The latent representations were then utilized as inputs to a deep neural network classifying BPE into BI-RADS classes. The network's predictions were elucidated at the radiomic feature level with Shapley values. The deep neural network achieved a BPE classification accuracy of 84 ± 2% (p-value < 0.00001). Most of the misclassifications involved adjacent classes. Different radiomic features were decisive for the prediction of each BPE class underlying the complexity of the decision boundaries. A highly precise and explainable pipeline for BPE classification was achieved without user- or algorithm-dependent radiomic feature selection.
Collapse
|
research-article |
1 |
|
25
|
Barabasch A, Hansen N, Lierfeld M, Ciritsis A, Kuhl C, Krämer N. Diffusionsgewichtete MRT vs. PET/CT zur Abschätzung des frühen Therapieansprechens nach SIRT. ROFO-FORTSCHR RONTG 2014. [DOI: 10.1055/s-0034-1372851] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
|
11 |
|