The Dejour classification for trochlear dysplasia shows slight interobserver and substantial intraobserver reliability

PURPOSE

Trochlear dysplasia is one of the main risk factors for recurrent patellar dislocation. The Dejour classification identifies four categories that can be used to classify trochlear dysplasia. The purpose of this study is to evaluate the inter- and intraobserver reliability of the Dejour classification for trochlear dysplasia. The hypothesis was that both intra- and interobserver reliability would be at least moderate.

METHODS

This is a cross-sectional, reliability study. Twenty-eight examiners from the International Patellofemoral Study Group 2022 meeting evaluated lateral radiographs of the knee and axial magnetic resonance images from 15 cases of patellofemoral instability with trochlear dysplasia. They classified each case according to Dejour's classification for trochlear dysplasia (A-D). There were three rounds: one with only computed radiograph (CR), one with only magnetic resonance imaging (MRI) and one with both. Inter- and intraobserver reliability were calculated using κ coefficient (0-1).

RESULTS

The mean age of patients was: 14.6 years; 60% were female and 53% had open physis. The interobserver reliability κ probabilities were 0.2 (CR), 0.13 (MRI) and 0.12 (CR and MRI). The intraobserver reliability κ probabilities were 0.45 (CR), 0.44 (MRI) and 0.65 (CR and MRI).

CONCLUSION

The Dejour classification for trochlear dysplasia has slight interobserver reliability and substantial intraobserver reliability.

LEVEL OF EVIDENCE

Level I.

Interobserver and Intraobserver Variability of Four Suprascapular Notch Classification Systems

INTRODUCTION

Knowledge of the morphology of the suprascapular notch is clinically beneficial in patients with suspected suprascapular nerve compression or palsy. Several classification systems have been proposed for the morphological classification of the suprascapular notch and its several anatomical variations. The purpose of this study was to evaluate the inter- and intraobserver reliability of four different classification systems for suprascapular notch typing analysing shoulder computed tomography (CT) scans.

METHODS

Shoulder CT scans from 109 subjects (71.5% males) were examined by three raters of various experience levels, one senior, one experienced, and one junior orthopaedic surgeon. The CT scans were evaluated quantitatively and qualitatively and the suprascapular notch was classified according to four classification systems at two separate timepoints, four weeks apart. To determine consistency among the same or different raters, the Kappa statistic was performed and intrarater reliability for each rater between the first and the second evaluation was assessed using Cohen's kappa. Reliability across all raters at each timepoint was assessed using the Fleiss kappa.

RESULTS

Agreement was almost perfect for all the classification systems and amongst all raters, regardless of their experience level. There were no significant differences between the raters on any of the evaluations. The overall interobserver agreement for all classifications was almost perfect.

CONCLUSION

The four suprascapular notch classification systems are reliable, and the rater's experience level has no impact on the evaluation.

Intraobserver Reliability on Classifying Bursitis on Shoulder Ultrasound

Purpose: Bursitis is a common musculoskeletal cause of shoulder pain and treatment varies, thus correctly diagnosing and grading bursitis is paramount in deciding management. Our aim was to assess reliability in grading shoulder bursitis on ultrasonography among fellowship trained musculoskeletal radiologists at our institution. Methods: Retrospective study of patients diagnosed with bursitis on ultrasonography. Single-sonographic images of the subacromial-subdeltoid bursa were collected for each patient and randomized to form a test-bank of varying degrees of bursitis. Three months after the test was administered, the cases were randomized and readministered. The radiologists graded each case as: within normal limits, mild, moderate or severe. Intraobserver variability was measured using Cohen's kappa coefficient. Linear regression model was performed to assess correlation between years of experience and kappa. Results: 10 radiologists reviewed 70 cases of bursitis. Kappa values ranged from .53 to .91, indicating 'moderate' to 'almost perfect' variability amongst radiologists. A moderate positive correlation of improving variability (r = .69) with increasing years of experience exists. Conclusion: Fellowship trained musculoskeletal radiologists were able to grade shoulder bursitis with moderate to almost perfect variability, with a positive correlation of improved variability with increasing experience. This may help clinicians choose the correct treatment more confidently in their patients with shoulder pain.

Intra- and Interobserver Variability in Magnetic Resonance Imaging Measurements in Rectal Cancer Patients

Colorectal cancer is the second most common cancer in Europe, and accurate lymph node staging in rectal cancer patients is essential for the selection of their treatment. MRI lymph node staging is complex, and few studies have been published regarding its reproducibility. This study assesses the inter- and intraobserver variability in lymph node size, apparent diffusion coefficient (ADC) measurements, and morphological characterization among inexperienced and experienced radiologists. Four radiologists with different levels of experience in MRI rectal cancer staging analyzed 36 MRI scans of 36 patients with rectal adenocarcinoma. Inter- and intraobserver variation was calculated using interclass correlation coefficients and Cohens-kappa statistics, respectively. Inter- and intraobserver agreement for the length and width measurements was good to excellent, and for that of ADC it was fair to good. Interobserver agreement for the assessment of irregular border was moderate, heterogeneous signal was fair, round shape was fair to moderate, and extramesorectal lymph node location was moderate to almost perfect. Intraobserver agreement for the assessment of irregular border was fair to substantial, heterogeneous signal was fair to moderate, round shape was fair to moderate, and extramesorectal lymph node location was substantial to almost perfect. Our data indicate that subjective variables such as morphological characteristics are less reproducible than numerical variables, regardless of the level of experience of the observers.

Intraobserver Variability in Bladder Cancer Treatment Response Assessment With and Without Computerized Decision Support

We evaluated the intraobserver variability of physicians aided by a computerized decision-support system for treatment response assessment (CDSS-T) to identify patients who show complete response to neoadjuvant chemotherapy for bladder cancer, and the effects of the intraobserver variability on physicians' assessment accuracy. A CDSS-T tool was developed that uses a combination of deep learning neural network and radiomic features from computed tomography (CT) scans to detect bladder cancers that have fully responded to neoadjuvant treatment. Pre- and postchemotherapy CT scans of 157 bladder cancers from 123 patients were collected. In a multireader, multicase observer study, physician-observers estimated the likelihood of pathologic T0 disease by viewing paired pre/posttreatment CT scans placed side by side on an in-house-developed graphical user interface. Five abdominal radiologists, 4 diagnostic radiology residents, 2 oncologists, and 1 urologist participated as observers. They first provided an estimate without CDSS-T and then with CDSS-T. A subset of cases was evaluated twice to study the intraobserver variability and its effects on observer consistency. The mean areas under the curves for assessment of pathologic T0 disease were 0.85 for CDSS-T alone, 0.76 for physicians without CDSS-T and improved to 0.80 for physicians with CDSS-T (P = .001) in the original evaluation, and 0.78 for physicians without CDSS-T and improved to 0.81 for physicians with CDSS-T (P = .010) in the repeated evaluation. The intraobserver variability was significantly reduced with CDSS-T (P < .0001). The CDSS-T can significantly reduce physicians' variability and improve their accuracy for identifying complete response of muscle-invasive bladder cancer to neoadjuvant chemotherapy.

Interobserver and Intraobserver Variability in the CT Assessment of COVID-19 Based on RSNA Consensus Classification Categories

PURPOSE

To assess the interobserver and intraobserver agreement of fellowship trained chest radiologists, nonchest fellowship-trained radiologists, and fifth-year radiology residents for COVID-19-related imaging findings based on the consensus statement released by the Radiological Society of North America (RSNA).

METHODS

A survey of 70 chest CTs of polymerase chain reaction (PCR)-confirmed COVID-19 positive and COVID-19 negative patients was distributed to three groups of participating radiologists: five fellowship-trained chest radiologists, five nonchest fellowship-trained radiologists, and five fifth-year radiology residents. The survey asked participants to broadly classify the findings of each chest CT into one of the four RSNA COVID-19 imaging categories, then select which imaging features led to their categorization. A 1-week washout period followed by a second survey comprised of randomly selected exams from the initial survey was given to the participating radiologists.

RESULTS

There was moderate overall interobserver agreement in each group (κ coefficient range 0.45-0.52 ± 0.02). There was substantial overall intraobserver agreement across the chest and nonchest groups (κ coefficient range 0.61-0.67 ± 0.06) and moderate overall intraobserver agreement within the resident group (κ coefficient 0.58 ± 0.06). For the image features that led to categorization, there were varied levels of agreement in the interobserver and intraobserver components that ranged from fair to perfect kappa values. When assessing agreement with PCR-confirmed COVID status as the key, we observed moderate overall agreement within each group.

CONCLUSION

Our results support the reliability of the RSNA consensus classification system for COVID-19-related image findings.

Reproducibility of Grading in Chronic Intervillositis of Unknown Etiology

Chronic intervillositis of unknown etiology (CIUE) is a rare placental inflammatory process associated with pregnancy loss and recurrence. We conducted a quality assurance study to assess the diagnostic accuracy and reproducibility of CIUE grading at our institution. Hematoxylin and eosin-stained slides from 20 CIUE cases (31 slides) were reviewed by 7 perinatal pathologists in 2 sequential rounds. Reviewers were instructed to use the diagnostic criteria they were presently following for CIUE and to grade each slide according to the Rota scheme. In the first round, 20 slides were assessed. The diagnostic accuracy was 94%, the average percent agreement of Rota grade was 79%, and the Fleiss' kappa value for interobserver variability was 0.54. The results were reviewed by all pathologists with diagnostic and grading criteria agreed upon prior to the second round. In round 2, the remaining 11 slides were assessed. Diagnostic accuracy was 83%, the average percent agreement on Rota grade was 70%, and the Fleiss' kappa value for interobserver variability was 0.36. Overall, diagnostic accuracy was high and agreement on Rota grade was moderate. Group review did not appear to improve accuracy. Simplifying CIUE grading to a low-grade/high-grade scheme (<50% or ≥50%) might improve grading reproducibility.

Intraobserver and Interobserver Variability in Ultrasound Measurements of Thyroid Nodules

OBJECTIVES

The purpose of this study was to assess the intraobserver and interobserver variability in ultrasound (US) measurements of thyroid nodules.

METHODS

We performed a prospective study of the US examinations of 73 patients with 122 thyroid nodules greater than 5 mm in size. Ultrasound measurements in 4 dimensions (anteroposterior, transverse, longitudinal, and maximum diameters) and measurement of the estimated volume (using the ellipsoid formula) of each thyroid nodule were performed twice by 2 independent radiologists (A and B, with 10 years and 6 months of experience, respectively). The intraobserver and interobserver variability in measurements of thyroid nodules was assessed by a Bland-Altman analysis of agreement. The absolute values for intraobserver and interobserver variability were compared by a paired t test.

RESULTS

The 95% intraobserver and interobserver limits of agreement for the anteroposterior, transverse, longitudinal, and maximum diameters and estimated volume of thyroid nodules were ±18.2%, ± 14.3%, and ±21.0%; ± 17.2%%, ± 17.3%, and 18.2%; ± 14.6%, ± 15.5%, and ±22.3%; ± 13.8%, ± 15.5%, and ±19.6%; and ±30.2%, ± 27.7%, and ±44.1%, respectively. The absolute values for intraobserver variability were lower than those for interobserver variability for all measurements.

CONCLUSIONS

There was considerable intraobserver and interobserver variability in US measurement of thyroid nodules, which must be taken into account during follow-up US examinations of patients with thyroid nodules.

Validation of diagnostic accuracy with whole-slide imaging compared with glass slide review in dermatopathology

BACKGROUND

Teledermatopathology has evolved from static images to whole slide imaging (WSI), which allows for remote viewing and manipulation of tissue sections. Previous studies of WSI in teledermatopathology predated College of American Pathologists (CAP) telepathology validation guidelines.

OBJECTIVE

We conducted a comprehensive retrospective WSI validation study of routine dermatopathology cases, adhering to CAP guidelines.

METHOD

In all, 181 consecutive cases arranged into 3 categories (inflammatory, melanocytic, nonmelanocytic proliferations) were reviewed by 3 board-certified dermatopathologists via traditional microscopy (TM) and WSI. Intraobserver (TM vs WSI), TM intraobserver and interobserver (TM vs TM), and WSI interobserver (WSI vs WSI) concordance was interpreted using a 3-tier system.

RESULTS

TM versus WSI intraobserver concordance (86.9%; 95% confidence interval [CI] 83.7-89.6) did not differ from TM versus TM intraobserver concordance (90.3%; 95% CI 86.7-93.1) or interobserver concordance (WSI: 89.9%; 95% CI 87.0-92.2, and TM: 89.5%; 95% CI 86.5-91.9). Melanocytic proliferations had the lowest TM versus WSI intraobserver concordance (75.6%; 95% CI 68.5-81.5), whereas inflammatory lesions had the highest TM versus WSI intraobserver concordance (96.1%; 95% CI 91.8-98.3). Nonmelanocytic proliferations had an intraobserver concordance of 89.1% (95% CI 83.4-93.0).

LIMITATIONS

Efficiency and other logistical WSI parameters were not evaluated.

CONCLUSION

Intraobserver and interobserver diagnostic concordance between WSI and TM was equivalent. Therefore, WSI appears to be a reliable diagnostic modality for dermatopathology.

Carotid inter-adventitial diameter is more strongly related to plaque score than lumen diameter: An automated tool for stroke analysis

PURPOSE

To compare the strength of correlation between automatically measured carotid lumen diameter (LD) and interadventitial diameter (IAD) with plaque score (PS).

METHODS

Retrospective study on a database of 404 common carotid artery B-mode sonographic images from 202 diabetic patients. LD and IAD were computed automatically using an advanced computerized edge detection method and compared with two distinct manual measurements. PS was computed by adding the maximal thickness in millimeters of plaques in segments taken from the internal carotid artery, bulb, and common carotid artery on both sides.

RESULTS

The coefficient of correlation was 0.19 (p < 0.007) between LD and PS, and 0.25 (p < 0.0006) between IAD and PS. After excluding 10 outliers, coefficient of correlation was 0.25 (p < 0.0001) between LD and PS, and 0.38 (p < 0.0001) between IAD and PS. The precision of merit of automated versus the two manual measurements was 96.6% and 97.2% for LD, and 97.7% and 98.1%, for IAD, respectively.

CONCLUSIONS

Our automated measurement system gave satisfying results in comparison with manual measurements. Carotid IAD was more strongly correlated to PS than carotid LD in this population sample of Japanese diabetic patients.

Thymic measurements in pathologically proven normal thymus and thymic hyperplasia: intraobserver and interobserver variabilities

RATIONALE AND OBJECTIVES

To determine the intraobserver and interobserver variabilities of thymic measurements on computed tomography (CT) in patients with pathologic diagnosis of thymic hyperplasia or normal thymus.

MATERIALS AND METHODS

Thirty-three patients with pathologic diagnosis of thymic hyperplasia (n = 25) or normal thymus (n = 8) who had identifiable thymus gland on CT were retrospectively studied. Two radiologists independently measured thymic size and CT attenuation. Concordance correlation coefficients (CCCs) and Bland-Altman plots were used to assess intraobserver and interobserver agreements.

RESULTS

The intraobserver and interobserver agreements of thymic diameters and the lobe length were moderate, with CCCs ranging from 0.73 to 0.89 and from 0.72 to 0.81, respectively. Higher agreement was noted among patients whose measurements were performed on the same CT image in two independent measurements, with intraobserver CCC ≥ 0.95 for diameters and length. After providing readers with an instruction for consistent selection of CT image for measurements, the intraobserver and interobserver agreements improved, resulting in CCCs ranging from 0.81 to 0.92 and from 0.77 to 0.85 for diameters and length, respectively. Thymic lobe thickness had the least agreement. CT attenuation measurements were highly reproducible, with CCCs ranging from 0.88 to 0.97. In patients with thymic CT attenuation >30 HU (Hounsfield unit), the attenuation measurements were more reproducible with narrower 95% limits of agreement.

CONCLUSIONS

Thymic size measurements had moderate-to-high intraobserver and interobserver agreements, when the instruction for consistent selection of images was provided to the readers. CT attenuation was highly reproducible, with higher reproducibility for thymic glands with >30 HU. Awareness of thymic measurement variability is necessary when interpreting measured values of normal thymus and thymic pathology on CT.

Intraobserver and interobserver variability in diagnosing scapholunate dissociation by cineradiography

PURPOSE

To evaluate the intraobserver and interobserver variability in diagnosing scapholunate dissociation (SLD) by wrist cineradiography.

METHODS

A musculoskeletal radiologist, hand surgeon, and trauma surgeon assessed the records of 50 consecutive wrist cineradiographies performed in 25 patients. Fluoroscopy was performed on the unaffected and affected wrist and consisted of radiographer-controlled passive flexion-extension and passive radial-ulnar deviation in both posteroanterior and lateral projections. To determine the intraobserver variability, the 3 reviewers reassessed all wrist cineradiographies 6 months after their first assessment. The kappa coefficient for interobserver agreement was calculated using the jackknife method. The Cohen kappa was used to assess intraobserver variability.

RESULTS

The interobserver variability for diagnosing SLD by cineradiography was excellent (κ = 0.84). The intraobserver variability for the hand surgeon was excellent (κ = 0.80), and was good for the radiologist (κ = 0.72) and the trauma surgeon (κ = 0.76).

CONCLUSIONS

Cineradiography is a promising and helpful, noninvasive tool for diagnosing SLD. It is widely available and has relatively low costs. Conventional radiographs remain essential in the primary workup for suspected SLD. However, we recommend cineradiography when an SLD is clinically suspected.

TYPE OF STUDY/LEVEL OF EVIDENCE

Diagnostic III.

Repeat measurements of nuchal translucency at 11-14 weeks of gestation: when do we need them?

OBJECTIVE

Previous studies have recommended that several repeat measurements of nuchal translucency (NT) be obtained to optimize sensitivity of the screening process. However, truncation is applied within the risk calculation for small NT measurements; therefore, repeating NT measurements in the lower range may be unnecessary. The aim of this study was to determine the optimal number of NT measurements and whether this is dependent on the initial value obtained.

METHODS

We simulated the expected distribution of sets of five repeat NT measurements and their corresponding likelihood ratios (LR) for a variety of crown-rump length (CRL) values at 11-14 weeks' gestation, based on the published mixture model for first-trimester NT measurements and on published data regarding NT measurement variability. The ratio between LR obtained from the highest and the initial measurement in each set were computed, with a ratio of 1 indicating that repeat measurements would have no effect on risk estimation. We calculated NT cut-offs below which a change in estimated LR would occur in fewer than 10% of cases if repeat measurements were obtained, and we tested this approach on a real first-trimester screening dataset.

RESULTS

The simulations performed indicated that repeating NT measurements when the first NT obtained ranged between ≤ 0.9 and ≤ 1.6 mm at CRLs of 45 and 84 mm, respectively, is not useful. When applied to a real dataset, our approach allowed avoiding repetition of measurements in 47/165 cases (28%).

CONCLUSIONS

Although there is some variability in NT measurements, repeat assessment is not useful to optimize screening performance for the smallest NT measurements due to lower truncation limits that are applied in risk calculation. Our study provides NT cut-off values, in relation to CRL, below which it appears that there is no need to repeat measurements once a good quality image has been obtained.

Intraobserver and interobserver variability in computed tomography size and attenuation measurements in patients with renal cell carcinoma receiving antiangiogenic therapy: implications for alternative response criteria

BACKGROUND

Alternative response criteria have been proposed in patients with metastatic renal cell carcinoma (mRCC) who are receiving vascular endothelial growth factor (VEGF)-targeted therapy, including 10% tumor shrinkage as an indicator of response/outcome. However, to the authors' knowledge, intraobserver and interobserver measurement variability have not been defined in this setting. The objective of the current study was to determine intraobserver and interobserver agreement of computed tomography (CT) size and attenuation measurements to establish reproducible response indicators.

METHODS

Seventy-one patients with mRCC with 179 target lesions were enrolled in phase 2 and phase 3 trials of VEGF-targeted therapies and retrospectively studied with Institutional Review Board approval. Two radiologists independently measured the long axis diameter and mean attenuation of target lesions at baseline and on follow-up CT. Concordance correlation coefficients and Bland-Altman plots were used to assess intraobserver and interobserver agreement.

RESULTS

High concordance correlation coefficients (range, 0.8602-0.9984) were observed in all types of measurements. The 95% limits of agreement for the percentage change of the sum longest diameter was -7.30% to 7.86% for intraobserver variability, indicating that 10% tumor shrinkage represents a true change in tumor size when measured by a single observer. The 95% limits of interobserver variability were -16.3% to 15.4%. On multivariate analysis, the location of the lesion was found to significantly contribute to interobserver variability (P = .048). The 95% limits of intraobserver agreement for the percentage change in CT attenuation were -18.34% to 16.7%.

CONCLUSIONS

In patients with mRCC who are treated with VEGF inhibitors, 10% tumor shrinkage is a reproducible radiologic response indicator when baseline and follow-up studies are measured by a single radiologist. Lesion location contributes significantly to measurement variability and should be considered when selecting target lesions.

Educating chiropractic students about intraobserver and interobserver variability through the use of skinfold measurement

PURPOSE

Skinfold measurements taken by novice observers are fraught with high rates of intraobserver variability and even higher rates of interobserver variability, and therefore having students collect and analyze skinfold measurements is an ideal way to present the concepts of both measurement and physiological variability.

METHODS

Students in a 1st trimester clinical biochemistry laboratory were assembled into groups of four, and within each group the students were asked to volunteer to be either a subject, a data recorder, or one of two observers. To demonstrate intraobserver versus interobserver variability the subject was assessed by four separate observers who each took four separate skinfold measurements. The skinfold measurement obtained from each separate observation was based on the sum of four skinfold sites. The average sums of the skinfold measurement (in mm) and standard deviations were calculated and posted for postlaboratory discussion.

RESULTS

Skinfold measurements were taken on 76 1st-trimester chiropractic students (46 males and 30 females). The average intraobserver and interobserver variability across all 76 participants was 4.8 +/- 2.3 mm and 10.0 +/- 6.3 mm, respectively, representing a twofold increase in variability, which was statistically significant (p < .0001). The noticeable differences between intraobserver and interobserver variability provided a great back drop for postlab discussion, which was the intended purpose of performing this demonstration project.

CONCLUSION

Measuring skinfold thickness can prove to be a useful and inexpensive method for easily and safely demonstrating the concepts of variability to students.

P16 and Ki67 Immunostains Decrease Intra- and Interobserver Variability in the Diagnosis and Grading of Anal Intraepithelial Neoplasia (AIN)

BACKGROUND

Significant variation is reported in the diagnosis of HPV-associated AIN. We previously observed that band-like positivity for p16 in >90% of contiguous cells coupled with Ki67 positivity in >50% of lesional cells is strongly associated with high grade AIN. This study was undertaken to determine if addition of p16 and Ki67 immunostaining would reduce inter- and intraobserver variability in diagnosis and grading of AIN.

DESIGN

H&E stained slides of 60 anal biopsies were reviewed by three pathologists and consensus diagnoses were achieved: 25 negative, 12 low (condyloma and/or AIN I) and 23 high (9 AIN II and 14 AIN III) grade lesions. The H&E stained slides were diagnosed independently by three additional ("participant") pathologists. Several weeks later they re-examined these slides in conjunction with corresponding p16 and Ki67 immunostains.

RESULTS

Addition of p16 and Ki67 immunostains reduced intra- and interobserver variability, improved concurrence with consensus diagnoses and reduced two-step differences in diagnosis. Negative and high grade AIN diagnoses showed the most improvement in concurrence levels.

CONCLUSION

Addition of p16 and Ki67 immunostains is helpful in the diagnosis and grading of AIN.