1
|
Guo D, Liu X, Wang D, Tang X, Qin Y. Development and clinical validation of deep learning for auto-diagnosis of supraspinatus tears. J Orthop Surg Res 2023; 18:426. [PMID: 37308995 DOI: 10.1186/s13018-023-03909-z] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/11/2023] [Accepted: 06/04/2023] [Indexed: 06/14/2023] Open
Abstract
BACKGROUND Accurately diagnosing supraspinatus tears based on magnetic resonance imaging (MRI) is challenging and time-combusting due to the experience level variability of the musculoskeletal radiologists and orthopedic surgeons. We developed a deep learning-based model for automatically diagnosing supraspinatus tears (STs) using shoulder MRI and validated its feasibility in clinical practice. MATERIALS AND METHODS A total of 701 shoulder MRI data (2804 images) were retrospectively collected for model training and internal test. An additional 69 shoulder MRIs (276 images) were collected from patients who underwent shoulder arthroplasty and constituted the surgery test set for clinical validation. Two advanced convolutional neural networks (CNN) based on Xception were trained and optimized to detect STs. The diagnostic performance of the CNN was evaluated according to its sensitivity, specificity, precision, accuracy, and F1 score. Subgroup analyses were performed to verify its robustness, and we also compared the CNN's performance with that of 4 radiologists and 4 orthopedic surgeons on the surgery and internal test sets. RESULTS Optimal diagnostic performance was achieved on the 2D model, from which F1-scores of 0.824 and 0.75, and areas under the ROC curves of 0.921 (95% confidence interval, 0.841-1.000) and 0.882 (0.817-0.947) were observed on the surgery and internal test sets. For the subgroup analysis, the 2D CNN model demonstrated a sensitivity of 0.33-1.000 and 0.625-1.000 for different degrees of tears on the surgery and internal test sets, and there was no significant performance difference between 1.5 and 3.0 T data. Compared with eight clinicians, the 2D CNN model exhibited better diagnostic performance than the junior clinicians and was equivalent to senior clinicians. CONCLUSIONS The proposed 2D CNN model realized the adequate and efficient automatic diagnoses of STs, which achieved a comparable performance of junior musculoskeletal radiologists and orthopedic surgeons. It might be conducive to assisting poor-experienced radiologists, especially in community scenarios lacking consulting experts.
Collapse
Affiliation(s)
- Deming Guo
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China
| | - Xiaoning Liu
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China
| | - Dawei Wang
- Beijing Infervision Technology Co Ltd, Beijing, People's Republic of China
| | - Xiongfeng Tang
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China.
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China.
| | - Yanguo Qin
- Orthopaedic Medical Center, The Second Hospital of Jilin University, Changchun, 130041, People's Republic of China.
- Jilin Provincial Key Laboratory of Orhtopeadics, Changchun, People's Republic of China.
| |
Collapse
|
2
|
Wang Z, Manassi M, Ren Z, Ghirardo C, Canas-Bajo T, Murai Y, Zhou M, Whitney D. Idiosyncratic biases in the perception of medical images. Front Psychol 2022; 13:1049831. [PMID: 36600706 PMCID: PMC9806180 DOI: 10.3389/fpsyg.2022.1049831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Accepted: 11/29/2022] [Indexed: 12/23/2022] Open
Abstract
Introduction Radiologists routinely make life-altering decisions. Optimizing these decisions has been an important goal for many years and has prompted a great deal of research on the basic perceptual mechanisms that underlie radiologists' decisions. Previous studies have found that there are substantial individual differences in radiologists' diagnostic performance (e.g., sensitivity) due to experience, training, or search strategies. In addition to variations in sensitivity, however, another possibility is that radiologists might have perceptual biases-systematic misperceptions of visual stimuli. Although a great deal of research has investigated radiologist sensitivity, very little has explored the presence of perceptual biases or the individual differences in these. Methods Here, we test whether radiologists' have perceptual biases using controlled artificial and Generative Adversarial Networks-generated realistic medical images. In Experiment 1, observers adjusted the appearance of simulated tumors to match the previously shown targets. In Experiment 2, observers were shown with a mix of real and GAN-generated CT lesion images and they rated the realness of each image. Results We show that every tested individual radiologist was characterized by unique and systematic perceptual biases; these perceptual biases cannot be simply explained by attentional differences, and they can be observed in different imaging modalities and task settings, suggesting that idiosyncratic biases in medical image perception may widely exist. Discussion Characterizing and understanding these biases could be important for many practical settings such as training, pairing readers, and career selection for radiologists. These results may have consequential implications for many other fields as well, where individual observers are the linchpins for life-altering perceptual decisions.
Collapse
Affiliation(s)
- Zixuan Wang
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
| | - Mauro Manassi
- School of Psychology, University of Aberdeen, King’s College, Aberdeen, United Kingdom
| | - Zhihang Ren
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
- Vision Science Group, University of California, Berkeley, Berkeley, CA, United States
| | - Cristina Ghirardo
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
| | - Teresa Canas-Bajo
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
- Vision Science Group, University of California, Berkeley, Berkeley, CA, United States
| | - Yuki Murai
- Center for Information and Neural Networks, National Institute of Information and Communications Technology, Koganei, Japan
| | - Min Zhou
- Department of Pediatrics, The First People's Hospital of Shuangliu District, Chengdu, Sichuan, China
| | - David Whitney
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
- Vision Science Group, University of California, Berkeley, Berkeley, CA, United States
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, United States
| |
Collapse
|
3
|
Alkaduhimi H, Saarig A, Amajjar I, van der Linde JA, van Wier MF, Willigenburg NW, van den Bekerom MPJ. Interobserver agreement for detecting Hill-Sachs lesions on magnetic resonance imaging. Clin Shoulder Elb 2021; 24:98-105. [PMID: 34078018 PMCID: PMC8181846 DOI: 10.5397/cise.2021.00115] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Accepted: 04/27/2021] [Indexed: 11/25/2022] Open
Abstract
BACKGROUND Our aim is to determine the interobserver reliability for surgeons to detect Hill-Sachs lesions on magnetic resonance imaging (MRI), the certainty of judgement, and the effects of surgeon characteristics on agreement. METHODS Twenty-nine patients with Hill-Sachs lesions or other lesions with a similar appearance on MRIs were presented to 20 surgeons without any patient characteristics. The surgeons answered questions on the presence of Hill-Sachs lesions and the certainty of diagnosis. Interobserver agreement was assessed using the Fleiss' kappa (κ) and percentage of agreement. Agreement between surgeons was compared using a technique similar to the pairwise t-test for means, based on large-sample linear approximation of Fleiss' kappa, with Bonferroni correction. RESULTS The agreement between surgeons in detecting Hill-Sachs lesions on MRI was fair (69% agreement; κ, 0.304; p<0.001). In 84% of the cases, surgeons were certain or highly certain about the presence of a Hill-Sachs lesion. CONCLUSIONS Although surgeons reported high levels of certainty for their ability to detect Hill-Sachs lesions, there was only a fair amount of agreement between surgeons in detecting Hill-Sachs lesions on MRI. This indicates that clear criteria for defining Hill-Sachs lesions are lacking, which hampers accurate diagnosis and can compromise treatment.
Collapse
Affiliation(s)
- Hassanin Alkaduhimi
- Shoulder and Elbow Unit, Joint Research, OLVG Hospital, Amsterdam, Netherlands
| | - Aïmane Saarig
- Shoulder and Elbow Unit, Joint Research, OLVG Hospital, Amsterdam, Netherlands
| | - Ihsan Amajjar
- Shoulder and Elbow Unit, Joint Research, OLVG Hospital, Amsterdam, Netherlands
| | - Just A van der Linde
- Department of Orthopedic Surgery and Traumatology, Reinier Haga Orthopedisch Centrum, Zoetermeer, Netherlands
| | - Marieke F van Wier
- Shoulder and Elbow Unit, Joint Research, OLVG Hospital, Amsterdam, Netherlands
| | | | | | | |
Collapse
|
4
|
Internal Derangement of the Shoulder Joint in Asymptomatic Professional Baseball Players. Acad Radiol 2020; 27:582-590. [PMID: 31300356 DOI: 10.1016/j.acra.2019.06.010] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2019] [Revised: 06/15/2019] [Accepted: 06/16/2019] [Indexed: 11/23/2022]
Abstract
RATIONALE AND OBJECTIVES To evaluate the influence of throwing activity on shoulder morphology and the difference in shoulder morphology on MRI between asymptomatic professional baseball players and volunteers who play baseball as a recreational activity. MATERIALS AND METHODS This retrospective case-control study included 68 asymptomatic professional baseball players (32 pitchers, 36 batters) and 30 male volunteers. Morphologic changes in the following shoulder structures were assessed on MRI: rotator cuff, glenoid labrum, humeral head, subacromial-subdeltoid bursa, subcoracoid bursa, long head of the biceps tendon, deltoid muscle, acromion, and clavicle. RESULTS Partially torn supraspinatus, posterior glenoid or labral lesions, bone marrow edema, intraosseous cysts of the humeral head, and edematous subacromial-subdeltoid bursa were significantly more commonly observed in players (p = 0.01, p < 0.001, p = 0.03, p< 0.001, and p < 0.001). Players with more than 10 years of experience had a significantly higher incidence of patchy intermediate signal abnormality (odds ratio: 3.73, p = 0.03), partial tear in the supraspinatus tendon (odds ratio: 6.20, p = 0.03), and edematous change in the subacromial-subdeltoid bursa (odds ratio: 2.96, p = 0.03). CONCLUSION The results from our study showed that repetitive throwing activities cause macroscopic structural lesions of the shoulder joints in asymptomatic baseball players. Significance of these lesions is to be determined.
Collapse
|
5
|
Annabell L, Master V, Rhodes A, Moreira B, Coetzee C, Tran P. Hip pathology: the diagnostic accuracy of magnetic resonance imaging. J Orthop Surg Res 2018; 13:127. [PMID: 29843749 PMCID: PMC5975565 DOI: 10.1186/s13018-018-0832-z] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/01/2018] [Accepted: 05/10/2018] [Indexed: 11/10/2022] Open
Abstract
Background Hip arthroscopy has led to a greater understanding of intra-articular hip pathology. Non-contrast magnetic resonance imaging (MRI) is currently the gold standard in non-invasive imaging diagnosis, with high sensitivity in identifying labral pathology but equivocal results for ligamentum teres damage and chondral defects. The aim of this study is to determine the accuracy of non-contrast MRI for diagnosis of intra-articular hip derangements and identify radiological features that could increase the accuracy of the diagnosis. Methods A prospective study of 71 hips on 68 patients undergoing hip arthroscopy was conducted comparing pre-operative analysis of MRI imaging versus an arthroscopic examination. Two musculoskeletal radiologists reported the data independently. All hip arthroscopies were performed by a single surgeon. Patients with MRIs performed within 6 months before hip arthroscopy were included. Outcome measures included observer accuracy identifying ligamentum teres tears, labral lesions, and chondral rim damage. Secondary outcome measures included inter-observer variability and correctly staged ligamentum teres tears. Results The accuracy of radiology reporting for ligamentum teres tears, labral damage, and chondral rim lesions was 85.92% for each instance. The MRI findings most consistent with labral tears include the presence of linear high signal traversing the articular surface into the labrum, presence of intra-labral fluid signal, and loss of homogenous low signal triangular morphology. Chondral rim damage was difficult to diagnose, but abnormal signal at the chondrolabral junction with partial thickness defects would suggest damage. Ligamentum teres tears are commonly found but poorly graded. Thickening and increased signal suggests synovitis while discontinuity and fraying suggests partial tearing. Conclusion Conventional non-arthrographic MRI offers an accurate non-invasive method to screen patients with symptoms referable to the hip by revealing the presence of labral tears, chondral defects, and ligamentum teres tears/synovitis. This study demonstrates that tears and synovitis of the ligamentum teres as potential sources of hip pain can be accurately identified on conventional non-arthrographic MRI. However, MRI has poor specificity and negative predictive value, and thus, a negative MRI result may warrant further investigation.
Collapse
Affiliation(s)
- Lucas Annabell
- Department of Orthopaedics, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia
| | - Vahid Master
- Department of Radiology, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia
| | - Alexander Rhodes
- Department of Radiology, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia
| | - Brett Moreira
- Department of Orthopaedics, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia
| | - Cassandra Coetzee
- Department of Orthopaedics, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia.
| | - Phong Tran
- Department of Orthopaedics, Western Health, 160 Gordon Street, Footscray, Melbourne, VIC, 3011, Australia
| |
Collapse
|
6
|
Etancelin-Jamet M, Bouilleau L, Martin A, Bertrand P. Diagnostic value of angled oblique sagittal images of the supraspinatus tendon for the detection of rotator cuff tears on MR imaging. Diagn Interv Imaging 2017; 98:161-169. [DOI: 10.1016/j.diii.2016.02.017] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2015] [Revised: 02/03/2016] [Accepted: 02/04/2016] [Indexed: 10/21/2022]
|
7
|
Carreño Mesa FA, Osma Rueda JL. Diagnóstico de la rotura del manguito de los rotadores (pruebas clínicas e imagenología). Revisión de conceptos actuales. ACTA ACUST UNITED AC 2016. [DOI: 10.1016/j.rccot.2016.09.009] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
|
8
|
van Grinsven S, van Loon C, van Gorp M, van Kints M, Konings P, van Kampen A. A feedback protocol improves the diagnostic performance of MR arthrography by experienced musculoskeletal radiologists in patients with traumatic anterior shoulder instability. Eur J Radiol 2015; 84:2242-9. [DOI: 10.1016/j.ejrad.2015.07.016] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2015] [Revised: 07/08/2015] [Accepted: 07/09/2015] [Indexed: 01/27/2023]
|
9
|
van Grinsven S, Nijenhuis TA, Konings PC, van Kampen A, van Loon CJM. Are radiologists superior to orthopaedic surgeons in diagnosing instability-related shoulder lesions on magnetic resonance arthrography? A multicenter reproducibility and accuracy study. J Shoulder Elbow Surg 2015; 24:1405-12. [PMID: 26175312 DOI: 10.1016/j.jse.2015.05.050] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2015] [Revised: 04/22/2015] [Accepted: 05/04/2015] [Indexed: 02/01/2023]
Abstract
BACKGROUND We compared the diagnostic reproducibility and accuracy of musculoskeletal radiologists with orthopaedic shoulder surgeons in 2 large medical centers in assessing magnetic resonance arthrograms (MRAs) of patients with traumatic anterior shoulder instability. METHODS Forty-five surgically confirmed MRAs were assessed by 4 radiologists, 4 orthopaedic surgeons, 2 radiologic teams, and 2 orthopaedic teams. During MRA assessment and surgery, the same 7-lesion scoring form was used. κ Coefficients, sensitivity, specificity, and differences in percentage of agreement or correct diagnosis (P < .05, McNemar test) were calculated per lesion and overall per the 7 lesion types. RESULTS The overall κ between the individual radiologists (κ = 0.51, κ = 0.46) and orthopaedic surgeons (κ = 0.46, κ = 0.41) was moderate. Although the overall percentage of agreement between the radiologists was slightly higher than that between the orthopaedic surgeons in both centers (80.0% vs 77.5% and 75.2% vs 73.7%), there was no significant difference. In each medical center, however, the most experienced orthopaedic surgeon was exceedingly more accurate than both radiologists per the 7 lesion types (81.9% vs 72.4%/74.6% and 76.5% vs 67.3%/73.7%). In 3 of 4 cases, this difference was significant. Overall accuracy improvement through consensus assessment was merely established for the weakest member of each team. CONCLUSION Experienced orthopaedic surgeons are more accurate than radiologists in assessing traumatic anterior shoulder instability-related lesions on MRA. In case of diagnosis disagreement, these orthopaedic surgeons should base their treatment decision on their own MRA interpretation.
Collapse
Affiliation(s)
- Susan van Grinsven
- Department of Orthopaedics, Rijnstate Hospital, Arnhem, The Netherlands.
| | - Thijs A Nijenhuis
- Department of Orthopaedics, Rijnstate Hospital, Arnhem, The Netherlands
| | - Peer C Konings
- Department of Orthopaedics, Rijnstate Hospital, Arnhem, The Netherlands
| | - Albert van Kampen
- Department of Orthopaedics, Radboud University Medical Centre, Nijmegen, The Netherlands
| | | |
Collapse
|
10
|
Kumar K, Makandura M, Leong NJJ, Gartner L, Lee CH, Ng DZW, Tan CH, Kumar VP. Is the Apprehension Test Sufficient for the Diagnosis of Anterior Shoulder Instability in Young Patients without Magnetic Resonance Imaging (MRI)? ANNALS OF THE ACADEMY OF MEDICINE, SINGAPORE 2015. [DOI: 10.47102/annals-acadmedsg.v44n5p178] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Introduction: The purpose of this study is to compare the sensitivities and positive predictive values (PPV) of the anterior apprehension test and magnetic resonance imaging (MRI) in the diagnosis of anterior labral tears in young patients with shoulder instability and to determine if surgery could be carried without this investigation in selected patients. Materials and Methods: We undertook a retrospective study of 168 patients aged between 15 and 30 years with a history of shoulder dislocation and compared the sensitivities and the PPV of the apprehension test with both MRI and magnetic resonance arthrograms (MRA) in the diagnosis of a Bankart lesion. The radiological investigations were interpreted by general practice radiologists and specialised musculoskeletal radiologists. All patients had their diagnosis confirmed by shoulder arthroscopy. Results: Our results showed that the apprehension test was highly reliable when it was positive with a PPV of 96%. It was more sensitive than MRI in the diagnosis of a Bankart lesion. The clinical test was significantly better when a musculoskeletal radiologist interpreted the MRI. The MRA interpreted by a musculoskeletal radiologist had the highest rates of sensitivity in detecting Bankart lesions. The figure was similar to that for the apprehension test. There was no difference in the PPVs among the clinical test, the MRI and the MRA read by the 2 categories of radiologists. Conclusion: We conclude that a routine MRI may be unnecessary in evaluating a young patient with clinically evident anterior shoulder instability if the apprehension test is positive. A MRA that can be interpreted by a musculoskeletal radiologist will be the next best investigation if the clinical test is negative or equivocal.
Key words: Dislocation, Glenohumeral joint, Radiological investigations
Collapse
Affiliation(s)
- Krishna Kumar
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| | - Milindu Makandura
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| | - Nicholas JJ Leong
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| | - Louise Gartner
- National University Hospital, National University Health System, Singapore
| | - Chin Hwee Lee
- National University Hospital, National University Health System, Singapore
| | - Dennis ZW Ng
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| | - Chyn Hong Tan
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| | - V Prem Kumar
- University Orthopaedics, Hand and Reconstructive Microsurgery Cluster, National University Hospital, National University Health System, Singapore
| |
Collapse
|
11
|
van Grinsven S, Hagenmaier F, van Loon CJM, van Gorp MJ, van Kints MJ, van Kampen A. Does the experience level of the radiologist, assessment in consensus, or the addition of the abduction and external rotation view improve the diagnostic reproducibility and accuracy of MRA of the shoulder? Clin Radiol 2014; 69:1157-64. [PMID: 25218253 DOI: 10.1016/j.crad.2014.07.009] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2014] [Revised: 07/05/2014] [Accepted: 07/09/2014] [Indexed: 12/21/2022]
Abstract
AIM To prospectively evaluate the influence of observer experience, consensus assessment, and abduction and external rotation (ABER) view on the diagnostic performance of magnetic resonance arthrography (MRA) in patients with traumatic anterior-shoulder instability (TASI). MATERIALS AND METHODS Fifty-eight MRA examinations (of which 51 had additional ABER views) were assessed by six radiologists (R1-R6) and three teams (T1-T3) with different experience levels, using a seven-lesion standardized scoring form. Forty-five out of 58 MRA examination findings were surgically confirmed. Kappa coefficients, sensitivity, specificity, and differences in percent agreement or correct diagnosis (p-value, McNemar's test) were calculated per lesion and overall per seven lesion types to assess diagnostic reproducibility and accuracy. RESULTS Overall kappa ranged from poor (k = 0.17) to moderate (k = 0.53), sensitivity from 30.6-63.5%, and specificity from 73.6-89.9%. Overall, the most experienced radiologists (R1-R2) and teams (T2-T3) agreed significantly more than the lesser experienced radiologists (R3-R4: p = 0.014, R5-R6; p = 0.018) and teams (T2-T3: p = 0.007). The most experienced radiologist (R1, R2, R3) and teams (T1, T2) were also consistently more accurate than the lesser experienced radiologists (R4, R5, R6) and team (T3). Significant differences were found between R1-R4 (p = 0.012), R3-R4 (p = 0.03), and T2-T3 (p = 0.014). The overall performance of consensus assessment was systematically higher than individual assessment. Significant differences were established between T1-T2 and radiologists R3-R4 (p<0.001, p = 0.001) and between T2 and R3 (p<0.001/p = 0.001) or R4 (p = 0.050). No overall significant differences were found between the radiologists' assessments with and without ABER. CONCLUSION The addition of ABER does not significantly improve overall diagnostic performance. The radiologist's experience level and consensus assessment do contribute to higher reproducibility and accuracy.
Collapse
Affiliation(s)
- S van Grinsven
- Department of Orthopaedics, Rijnstate Hospital, PO Box 9555, 6800 TA, Arnhem, The Netherlands.
| | - F Hagenmaier
- Department of Orthopaedics, Rijnstate Hospital, PO Box 9555, 6800 TA, Arnhem, The Netherlands
| | - C J M van Loon
- Department of Orthopaedics, Rijnstate Hospital, PO Box 9555, 6800 TA, Arnhem, The Netherlands
| | - M J van Gorp
- Department of Radiology, Rijnstate Hospital, PO Box 9555, 6800 TA, Arnhem, The Netherlands
| | - M J van Kints
- Department of Radiology, Rijnstate Hospital, PO Box 9555, 6800 TA, Arnhem, The Netherlands
| | - A van Kampen
- Department of Orthopaedics, Radboud University Medical Centre, PO Box 9101, 6501 HB, Nijmegen, The Netherlands
| |
Collapse
|
12
|
Lenza M, Buchbinder R, Takwoingi Y, Johnston RV, Hanchard NCA, Faloppa F. Magnetic resonance imaging, magnetic resonance arthrography and ultrasonography for assessing rotator cuff tears in people with shoulder pain for whom surgery is being considered. Cochrane Database Syst Rev 2013; 2013:CD009020. [PMID: 24065456 PMCID: PMC6464715 DOI: 10.1002/14651858.cd009020.pub2] [Citation(s) in RCA: 90] [Impact Index Per Article: 8.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
BACKGROUND Shoulder pain is a very common symptom. Disorders of the rotator cuff tendons due to wear or tear are among the most common causes of shoulder pain and disability. Magnetic resonance imaging (MRI), magnetic resonance arthrography (MRA) and ultrasound (US) are increasingly being used to assess the presence and size of rotator cuff tears to assist in planning surgical treatment. It is not known whether one imaging method is superior to any of the others. OBJECTIVES To compare the diagnostic test accuracy of MRI, MRA and US for detecting any rotator cuff tears (i.e. partial or full thickness) in people with suspected rotator cuff tears for whom surgery is being considered. SEARCH METHODS We searched the Cochrane Register of Diagnostic Test Accuracy Studies, MEDLINE, EMBASE, and LILACS from inception to February 2011. We also searched trial registers, conference proceedings and reference lists of articles to identify additional studies. No language or publication restrictions were applied. SELECTION CRITERIA We included all prospective diagnostic accuracy studies that assessed MRI, MRA or US against arthroscopy or open surgery as the reference standard, in people suspected of having a partial or full thickness rotator cuff tear. We excluded studies that selected a healthy control group, or participants who had been previously diagnosed with other specific causes of shoulder pain such as osteoarthritis or rheumatoid arthritis. Studies with an excessively long period (a year or longer) between the index and reference tests were also excluded. DATA COLLECTION AND ANALYSIS Two review authors independently extracted data on study characteristics and results of included studies, and performed quality assessment according to QUADAS criteria. Our unit of analysis was the shoulder. For each test, estimates of sensitivity and specificity from each study were plotted in ROC space and forest plots were constructed for visual examination of variation in test accuracy. Meta-analyses were performed using the bivariate model to produce summary estimates of sensitivity and specificity. We were unable to formally investigate potential sources of heterogeneity because of the small number of studies. MAIN RESULTS We included 20 studies of people with suspected rotator cuff tears (1147 shoulders), of which six evaluated MRI and US (252 shoulders), or MRA and US (127 shoulders) in the same people. Many studies had design flaws, with the potential for bias, thus limiting the reliability of their findings. Overall, the methodological quality of the studies was judged to be low or unclear. For each test, we observed considerable heterogeneity in study results, especially between studies that evaluated US for the detection of full thickness tears and studies that evaluated MRA for the detection of partial thickness tears. The criteria for a positive diagnostic test (index tests and reference standard) varied between studies.Meta-analyses were not possible for studies that assessed MRA for detection of any rotator cuff tears or partial thickness tears. We found no statistically significant differences in sensitivity or specificity between MRI and US for detecting any rotator cuff tears (P = 0.13), or for detecting partial thickness tears (P = 1.0). Similarly, for the comparison between MRI, MRA and US for detecting full thickness tears, there was no statistically significant difference in diagnostic performance (P = 0.7). For any rotator cuff tears, the summary sensitivity and specificity were 98% (95% CI 92% to 99%) and 79% (95% CI 68% to 87%) respectively for MRI (6 studies, 347 shoulders), and 91% (95% CI 83% to 95%) and 85% (95% CI 74% to 92%) respectively for US (13 studies, 854 shoulders). For full thickness tears, the summary sensitivity and specificity were 94% (95% CI 85% to 98%) and 93% (95% CI 83% to 97%) respectively for MRI (7 studies, 368 shoulders); 94% (95% CI 80% to 98%) and 92% (95% CI 83% to 97%) respectively for MRA (3 studies, 183 shoulders); and 92% (95% CI 82% to 96%) and 93% (95% CI 81% to 97%) respectively for US (10 studies, 729 shoulders).Because few studies were direct head-to-head comparisons, we could not perform meta-analyses restricted to these studies. The test comparisons for each of the three classifications of the target condition were therefore based on indirect comparisons which may be prone to bias due to confounding. AUTHORS' CONCLUSIONS MRI, MRA and US have good diagnostic accuracy and any of these tests could equally be used for detection of full thickness tears in people with shoulder pain for whom surgery is being considered. The diagnostic performance of MRI and US may be similar for detection of any rotator cuff tears. However, both MRI and US may have poor sensitivity for detecting partial thickness tears, and the sensitivity of US may be much lower than that of MRI. The strength of evidence for all test comparisons is limited because most studies were small, heterogeneous and methodologically flawed, and there were few comparative studies. Well designed studies that directly compare MRI, MRA and US for detection of rotator cuff tears are needed.
Collapse
Affiliation(s)
- Mário Lenza
- Hospital Israelita Albert EinsteinOrthopaedic and Trauma DepartmentAv. Albert Einstein, 627/701São PauloSao PauloBrazilCEP 05651‐901
| | - Rachelle Buchbinder
- Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash UniversityMonash Department of Clinical Epidemiology, Cabrini HospitalSuite 41, Cabrini Medical Centre183 Wattletree RoadMalvernVictoriaAustralia3144
| | - Yemisi Takwoingi
- University of BirminghamPublic Health, Epidemiology and BiostatisticsEdgbastonBirminghamUKB15 2TT
| | - Renea V Johnston
- Department of Epidemiology and Preventive Medicine, School of Public Health and Preventive Medicine, Monash UniversityMonash Department of Clinical Epidemiology, Cabrini HospitalSuite 41, Cabrini Medical Centre183 Wattletree RoadMalvernVictoriaAustralia3144
| | - Nigel CA Hanchard
- Teesside UniversityHealth and Social Care InstituteMiddlesbroughTees ValleyUKTS1 3BA
| | - Flávio Faloppa
- Universidade Federal de São PauloDepartment of Orthopaedics and TraumatologyRua Borges Lagoa, 783‐5th FloorSão PauloSão PauloBrazil
| | | |
Collapse
|
13
|
|
14
|
Nazarian LN, Jacobson JA, Benson CB, Bancroft LW, Bedi A, McShane JM, Miller TT, Parker L, Smith J, Steinbach LS, Teefey SA, Thiele RG, Tuite MJ, Wise JN, Yamaguchi K. Imaging algorithms for evaluating suspected rotator cuff disease: Society of Radiologists in Ultrasound consensus conference statement. Radiology 2013; 267:589-95. [PMID: 23401583 DOI: 10.1148/radiol.13121947] [Citation(s) in RCA: 63] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
Abstract
The Society of Radiologists in Ultrasound convened a panel of specialists from a variety of medical disciplines to reach a consensus about the recommended imaging evaluation of painful shoulders with clinically suspected rotator cuff disease. The panel met in Chicago, Ill, on October 18 and 19, 2011, and created this consensus statement regarding the roles of radiography, ultrasonography (US), computed tomography (CT), CT arthrography, magnetic resonance (MR) imaging, and MR arthrography. The consensus panel consisted of two co-moderators, a facilitator, a statistician and health care economist, and 10 physicians who have specialty expertise in shoulder pain evaluation and/or treatment. Of the 13 physicians on the panel, nine were radiologists who were chosen to represent a broad range of skill sets in diagnostic imaging, different practice types (private and academic), and different geographical regions of the United States. Five of the radiologists routinely performed musculoskeletal US as part of their practice and four did not. There was also one representative from each of the following clinical specialties: rheumatology, physical medicine and rehabilitation, orthopedic surgery, and nonoperative sports medicine. The goal of this conference was to construct several algorithms with which to guide the imaging evaluation of suspected rotator cuff disease in patients with a native rotator cuff, patients with a repaired rotator cuff, and patients who have undergone shoulder replacement. The panel hopes that these recommendations will lead to greater uniformity in rotator cuff imaging and more cost-effective care for patients suspected of having rotator cuff abnormality.
Collapse
Affiliation(s)
- Levon N Nazarian
- Department of Radiology, Thomas Jefferson University Hospital, 763E Main Building, 132 S Tenth St, Philadelphia, PA 19107-5244, USA.
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
15
|
Perdikakis E, Drakonaki E, Maris T, Karantanas A. MR arthrography of the shoulder: tolerance evaluation of four different injection techniques. Skeletal Radiol 2013; 42:99-105. [PMID: 23064511 DOI: 10.1007/s00256-012-1526-y] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/09/2012] [Revised: 08/30/2012] [Accepted: 09/17/2012] [Indexed: 02/02/2023]
Abstract
OBJECTIVE We sought to prospectively evaluate patients' pain perception and technical success of four different arthrographic techniques for shoulder MR arthrography. MATERIALS AND METHODS A total of 125 consecutive patients were referred for shoulder MR arthrography. The patients were randomly injected under fluoroscopic guidance (n(1) = 37), with CT guidance using an anterior (n(2) = 29) or a posterior approach (n(3) = 32) and with ultrasound guidance (n(4) = 27). For each patient, absolute periprocedural pain on a numerical rating pain scale (0 = "no pain", 10 = "intolerable pain"), technical success of the method used, and reason for referral were recorded. RESULTS The technical success rate was 100% for all injection methods. The results regarding absolute periprocedural pain were as follows: fluoroscopic guidance showed a mean pain of 4.05 ± 1.24, CT anterior guidance demonstrated a mean pain of 3.87 ± 0.95, CT posterior guidance showed a mean pain of 1.59 ± 0.81, and ultrasound guidance a mean pain of 3.63 ± 1.12. A significant difference (p < .05) was observed for the posterior route under CT guidance. The mean pain level was significantly higher for older (> 51 year) female patients. CONCLUSIONS No differences were found for the technical success rate of the aforementioned techniques. A CT-guided posterior approach seems to be a more comfortable method for the patient.
Collapse
Affiliation(s)
- Evangelos Perdikakis
- Department of Medical Imaging, University Hospital, Heraklion, 71110, Crete, Greece
| | | | | | | |
Collapse
|
16
|
Garavaglia G, Ufenast H, Taverna E. The frequency of subscapularis tears in arthroscopic rotator cuff repairs: A retrospective study comparing magnetic resonance imaging and arthroscopic findings. INTERNATIONAL JOURNAL OF SHOULDER SURGERY 2012; 5:90-4. [PMID: 22223958 PMCID: PMC3249929 DOI: 10.4103/0973-6042.91000] [Citation(s) in RCA: 65] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
PURPOSE With the advent of arthroscopic shoulder surgery the comprehension and description of rotator cuff tears have strongly evolved. Subscapularis tears are difficult to recognize and are underestimated. Our purpose is to report our observations concerning the relative frequency of subscapularis tears in patients undergoing arthroscopic rotator cuff repair and to compare the arthroscopic observations with the magnetic resonance imaging (MRI) findings. MATERIALS AND METHODS Retrospective cohort study including all patients undergoing arthroscopic rotator cuff repair was performed between March 2006 and March 2009 at our institution. Preoperative MRI findings, intraoperative arthroscopic findings, and details of surgical intervention were collected using medical charts. RESULTS We reviewed the medical charts of a total of 348 consecutive arthroscopic rotator cuff repairs. There were 311 supraspinatus tears (89%), 48 infraspinatus tears (14%), and 129 subscapularis tears (37%). MRI sensitivity and specificity were respectively 0.25 and 0.98 for subscapularis tendon tears, 0.67 and 1.0 for supraspinatus tears and 0.5 and 0.99 for infraspinatus tears. CONCLUSION Subscapularis tears are frequent lesions and usually appear concomitantly with supra or infraspinatus lesions. We propose a classification of subscapularis tendon tears, based on our observations of the pathoanatomy of the tears. While concordance with MRI results are good for the supraspinatus, MRI often fails to diagnose the presence of subscapularis tears and infraspinatus tears.
Collapse
|
17
|
A meta-analysis of the diagnostic test accuracy of MRA and MRI for the detection of glenoid labral injury. Arch Orthop Trauma Surg 2012; 132:905-19. [PMID: 22395821 DOI: 10.1007/s00402-012-1493-8] [Citation(s) in RCA: 77] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/03/2011] [Indexed: 12/17/2022]
Abstract
PURPOSE Magnetic resonance imaging (MRI) and magnetic resonance arthrography (MRA) have gained increasing favour in the assessment of patients with suspected glenoid labral injuries. The purpose of this study was to determine the diagnostic accuracy of MRI or MRA in the detection of gleniod labral lesions. MATERIALS AND METHODS A systematic review was undertaken of the electronic databases Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, AMED and CINAHL, in addition to a search of unpublished literature databases. All studies which compared the ability of MRI or MRA (index test) to assess gleniod labral tears or lesions, when verified with a surgical procedure (arthroscopy or open surgery-reference test) were included. Data extraction and methodological appraisal using the QUADAS tool were both conducted by two reviewers independently. Data were analysed through a summary receiver operator characteristic curve and pooled sensitivity and specificity analysis were calculated with 95% confidence intervals. RESULTS Sixty studies including 4,667 shoulders from 4,574 patients were reviewed. There appeared slightly greater diagnostic test accuracy for MRA over MRI for the detection of overall gleniod labral lesions (MRA-sensitivity 88%, specificity 93% vs. MRI sensitivity 76% vs. specificity 87%). Methodologically, studies recruited and identified their samples appropriately and clearly defined the radiological procedures. In general, it was not clearly defined why patients were lost during the study, and studies were poor at recording whether the same clinical data were available to the radiologist interpreting the MRI or MRA as would be available in clinical practice. Most studies did not state whether the surgeon interpreting the arthroscopic procedure was blinded to the results of the MR or MRA imaging. CONCLUSIONS Based on the available literature, overall MRA appeared marginally superior to MRI for the detection of glenohumeral labral lesions. LEVEL OF EVIDENCE Level 2a.
Collapse
|
18
|
Jonas SC, Walton MJ, Sarangi PP. Is MRA an unnecessary expense in the management of a clinically unstable shoulder? A comparison of MRA and arthroscopic findings in 90 patients. Acta Orthop 2012; 83:267-70. [PMID: 22401678 PMCID: PMC3369153 DOI: 10.3109/17453674.2012.672090] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/31/2023] Open
Abstract
BACKGROUND AND PURPOSE In detection of glenoid labrum pathology, MR arthrography (MRA) has shown sensitivities of 88-100% and specificities of 89-93%. However, our practice suggested that there may be a higher frequency of falsely negative reports. We assessed the accuracy of this costly modality in practice. PATIENTS AND METHODS We retrospectively reviewed MRA reports of 90 consecutive patients with clinical shoulder instability who had undergone shoulder arthroscopy. All had a history of traumatic anterior shoulder dislocation and had positive anterior apprehension tests. All underwent arthroscopy and stabilization during the same procedure. We compared the findings, using arthroscopic findings as the gold standard in the identification of glenoid labrum pathology. RESULTS 83 of the 90 patients had glenoid labrum tears at arthroscopy. Only 54 were correctly identified at MRA. All normal glenoid labra were identified at MRA. This gave a sensitivity of 65% and a specificity of 100% in identification of all types of glenoid labrum tear. 74 patients had anterior glenoid labral tears that were detected at an even lower rate of sensitivity (58%). INTERPRETATION The sensitivity of MRA in this series was substantially lower than previously published, suggesting that MRA may not be as reliable a diagnostic imaging modality in glenohumeral instability as previously thought. Our findings highlight the importance of an accurate history and clinical examination in the management of glenohumeral instability. The need for MRA may not be as high as is currently believed.
Collapse
Affiliation(s)
- Sam C Jonas
- Department of Orthopaedics, University Hospital Bristol, Bristol, UK.
| | | | | |
Collapse
|
19
|
McGuire CM, MacMahon P, Byrne DP, Kavanagh E, Mulhall KJ. Diagnostic accuracy of magnetic resonance imaging and magnetic resonance arthrography of the hip is dependent on specialist training of the radiologist. Skeletal Radiol 2012; 41:659-65. [PMID: 21912884 DOI: 10.1007/s00256-011-1266-4] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/19/2011] [Revised: 08/16/2011] [Accepted: 08/21/2011] [Indexed: 02/02/2023]
Abstract
OBJECTIVE Significant differences between magnetic resonance imaging reports and intraoperative findings at the time of hip arthroscopy were documented in our practice. We sought to examine the accuracy of radiological reporting of hip pathology based on the training level of the reporting radiologist. MATERIALS AND METHODS A retrospective review of hip arthroscopies carried out between July 2008 and June 2009 identified 61 cases where original MRI scans had been reported by general community radiologists. These scans were then reviewed by musculoskeletal specialist radiologists who were blinded to both the original report and the surgical findings. Accuracy of both subsets of radiologists was compared to arthroscopic findings with regard to labral, acetabular, femoral and impingement lesions. RESULTS Musculoskeletal radiologists performed better than community radiologists in terms of overall accuracy. Accuracy rates for MSK radiologists were 85, 79, 59, and 82% for labral, acetabular chondrosis, and femoral chondrosis and impingement lesions, respectively. Whereas accuracy rates for community radiologists were 70, 28, 52, and 59% (p values = 0.08, <0.001, 0.59, <0.001). Accuracy was significantly improved for both groups of radiologists when MR arthrograms were reviewed rather than conventional MRIs. CONCLUSIONS This study establishes the relationship between accuracy of reporting and the training level of the performing radiologists.
Collapse
Affiliation(s)
- Ciara M McGuire
- Department of Orthopaedics, Sports Surgery Clinic, Santry, Dublin, Ireland.
| | | | | | | | | |
Collapse
|
20
|
Ultrasonography Applications in Diagnosis and Management of Early Rheumatoid Arthritis. Rheum Dis Clin North Am 2012; 38:259-75. [DOI: 10.1016/j.rdc.2012.05.006] [Citation(s) in RCA: 11] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
21
|
Halma JJ, Eshuis R, Krebbers YMJ, Weits T, de Gast A. Interdisciplinary inter-observer agreement and accuracy of MR imaging of the shoulder with arthroscopic correlation. Arch Orthop Trauma Surg 2012; 132:311-20. [PMID: 21842285 PMCID: PMC3282007 DOI: 10.1007/s00402-011-1370-x] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/05/2010] [Indexed: 01/02/2023]
Abstract
INTRODUCTION Differences between radiologists and orthopaedic surgeons in the interpretation of MR images of the shoulder joint are experienced in daily clinical practice. This study set out to evaluate the inter-observer agreement between radiologists and orthopaedic surgeons in assessing pathology on MR imaging of the shoulder joint. Also, we determined the accuracy of the observers with arthroscopy as the standard of reference. MATERIALS AND METHODS Two radiologists and one orthopaedic surgeon reviewed 50 MR studies-25 conventional MR examinations and 25 MR arthrographies-of patients with shoulder complaints who had undergone MR imaging and subsequently arthroscopic surgery. The assessment was independent and blinded. All observers evaluated the MR examinations twice. Standard evaluation forms were used to score for pathology of rotator cuff, glenoid labrum, tendon of the long head of the biceps brachii and glenohumeral ligaments. The presence or absence of osteoarthritis, SLAP lesions, Bankart lesions, Hill-Sachs lesions or impingement was also noted. Intra- and inter-observer agreement, the sensitivity and specificity were calculated. Differences in percentages of correctly diagnosed lesions were tested for significance using McNemar's test. RESULTS There was a poor inter-observer agreement between the orthopaedic surgeon and the radiologists in assessing Bankart lesions and ligamentous lesions. We found significant differences between the radiologists and the orthopaedic surgeon in the assessment of osteoarthritis, Hill-Sachs lesions and impingement. CONCLUSION The orthopaedic surgeon and radiologists differed in their interpretation of what defines a Bankart lesion and what defines a ligamentous lesion. The orthopaedic surgeon was significantly more accurate in assessing impingement.
Collapse
Affiliation(s)
- J. J. Halma
- Clinical Orthopaedic Research Center, Department of Orthopaedics, Diakonessenhuis Hospital, Utrecht/Zeist, Bosboomstraat 1, 3582 KE Utrecht, The Netherlands ,Present Address: Burg. Reigerstraat 5, 3581 KJ Utrecht, The Netherlands
| | - R. Eshuis
- Clinical Orthopaedic Research Center, Department of Orthopaedics, Diakonessenhuis Hospital, Utrecht/Zeist, Bosboomstraat 1, 3582 KE Utrecht, The Netherlands
| | - Y. M. J. Krebbers
- Department of Radiology, Diakonessenhuis Hospital, Utrecht/Zeist, Bosboomstraat 1, 3582 KE Utrecht, The Netherlands
| | - T. Weits
- Department of Radiology, Diakonessenhuis Hospital, Utrecht/Zeist, Bosboomstraat 1, 3582 KE Utrecht, The Netherlands
| | - A. de Gast
- Clinical Orthopaedic Research Center, Department of Orthopaedics, Diakonessenhuis Hospital, Utrecht/Zeist, Bosboomstraat 1, 3582 KE Utrecht, The Netherlands ,Postbus 80250, 3508 TG Utrecht, The Netherlands
| |
Collapse
|