Agreement studies in radiology research

Automated grading system for quantifying KOH microscopic images in dermatophytosis

Concerning the progression of dermatophytosis and its prognosis, quantification studies play a significant role. Present work aims to develop an automated grading system for quantifying fungal loads in KOH microscopic images of skin scrapings collected from dermatophytosis patients. Fungal filaments in the images were segmented using a U-Net model to obtain the pixel counts. In the absence of any threshold value for pixel counts to grade these images as low, moderate, or high, experts were assigned the task of manual grading. Grades and corresponding pixel counts were subjected to statistical procedures involving cumulative receiver operating characteristic curve analysis for developing an automated grading system. The model's specificity, accuracy, precision, and sensitivity metrics crossed 92%, 86%, 82%, and 76%, respectively. 'Almost perfect agreement' with Fleiss kappa of 0.847 was obtained between automated and manual gradings. This pixel count-based grading of KOH images offers a novel, cost-effective solution for quantifying fungal load.

Radiological anatomical measurements of the superficial temporal artery: an interobserver reproducibility study

PURPOSE

Numerous studies have already reported on the anatomy of the superficial temporal artery (STA) on dissection or imaging, and more specifically on CT angiography. However, few studies have reported on the possible variations that exist between readers in the assessment of certain anatomical parameters. The aim of this study was to investigate the reproducibility of different anatomical measurements of the superficial temporal artery with CT angiography.

METHODS

Forty cranial iodine contrast-enhanced CT scans were searched in a retrospective monocentric study. Various anatomical categorical and quantitative measurements (more or less automatic) of the superficial temporal artery were assessed. To evaluate the inter-observer reproducibility of various anatomical measurements of the superficial temporal artery obtained through CT angiography, descriptive statistics, kappa statistics and intraclass correlation coefficients (ICC) were employed.

RESULTS

Eighty STAs were analyzed independently by two readers. The kappa coefficient for determining the mode of STA termination was substantial (kappa = 0.782). The number of loops was mean 0.22 vs. 0.25 (p = 0.981; ICC = 0.701) and the number of kinks was mean 5.68vs5.64 (p = 0.876; ICC = 0.860). No significant difference was found between readers for the determination of the distance of the STA termination to different superficial landmark lines. No significant difference in the orientation angles of the STA and the frontal and parietal branches was found. Semi-automatic measurements of the length of the STA appeared (ICC > 0.7) reproducible whereas semi-automatic measurements (surface, perimeter, short diameter, large diameter) appeared statistically less reproducible (ICC < 0.5). These results must also be interpreted in the light of the spatial resolutions inherent in the CT angiography technique.

CONCLUSION

The present study provides information concerning the reproducibility of certain anatomical measurements of the superficial temporal artery in CT angiography.

Meta-Analysis of Interobserver Agreement in Assessment of Interstitial Lung Disease Using High-Resolution CT

Background High-resolution CT (HRCT) is central to the assessment of interstitial lung disease (ILD), and accurate classification of disease has important implications for patients. Evaluation of imaging features can be challenging, even for experienced thoracic radiologists. Previous work has provided equivocal evidence on the interpretation of HRCT features at ILD-related imaging. Purpose To perform a meta-analysis to assess the level of agreement among expert thoracic radiologists in interpreting ILD-related imaging. Materials and Methods A systematic literature search from January 2000 to October 2023 of the Ovid MEDLINE, Embase, and Cochrane Central Register of Controlled Trials databases was performed for articles reporting assessments of interobserver agreement between thoracic radiologists for evaluation of ILD findings, such as severity and progression of disease, presence of features such as honeycombing and ground-glass opacification, and classification based on the 2011 and 2018 American Thoracic Society/European Respiratory Society/Japanese Respiratory Society/Asociación Latinoamericana del Tórax (ATS/ERS/JRS/ALAT) guidelines for idiopathic pulmonary fibrosis (IPF). Meta-analysis was performed using a random-effects model to obtain pooled κ or intraclass correlation coefficient (ICC) values as measures of interobserver agreement. Results The final analysis included 13 studies consisting of 6943 images and 146 radiologists. In 10 studies assessing agreement of specific radiologic findings in ILD, the pooled κ value was 0.56 (95% CI: 0.43, 0.70). In eight studies, the assessed interobserver agreement of the ATS/ERS/JRS/ALAT diagnostic guidelines for IPF based on usual interstitial pneumonia (UIP) patterns, the pooled κ value was 0.61 (95% CI: 0.48, 0.74). One study reported a κ value of 0.87 for ILD progression. Seven studies assessing ILD severity could not be pooled; the individual κ values for ILD severity ranged from 0.64 to 0.90, and ICC values ranged from 0.63 to 0.96. Conclusion There was moderate agreement between thoracic radiologists when assessing ILD features and UIP pattern diagnosis but little evidence on agreement of disease severity, extent, or progression. Meta-analysis registry no. PROSPERO CRD42022361803 © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Humbert in this issue.

Craniocervical Morphometry in Pomeranians-Part I: Intra-Observer, Interobserver, and Intermodality (CT vs. MRI) Agreement

BACKGROUND

Currently, there are no diagnostic imaging-based studies that have focused specifically on the craniocervical morphology of Pomeranian dogs in relation to Chiari-like malformation and syringomyelia (CM/SM). The aims of Part I of this two-part study are to assess the intra-observer, interobserver, and intermodality reliability and agreement for various aspects of the craniocervical morphology of Pomeranians with and without CM/SM.

METHODS

Prospectively, Pomeranians were included that underwent both CT and MRI studies of the head and cervicothoracic vertebral column. Two observers (experienced and novice) independently performed CM and SM classifications, qualitative assessments, and quantitative measurements. The experienced observer performed these assessments twice. A third observer (experienced) performed CM and SM normal or abnormal classifications.

RESULTS

Ninety-nine (99) dogs were included. Interobserver reliability was influenced by observer experience level. For the experienced versus novice observers, substantial interobserver agreement was found for classification of SM as normal or abnormal (Cohen's kappa = 0.63), while interobserver agreement was fair for classification of SM as normal or abnormal (Cohen's kappa = 0.31). Interobserver, intra-observer, and intermodality reliability were variable for different measurements and assessments, and best overall for the CT-based measurements.

CONCLUSIONS

Interobserver reliability and agreement results should be taken into account in the evaluation of results of future studies as well as the evaluation of imaging studies of Pomeranians presented clinically.

Observer agreement in single computerized tomography use for diagnosing paediatric head and neck malignancies at Uganda Cancer Institute

BACKGROUND

In the Ugandan setting, investigation for PHNM with CT uses a protocol with both unenhanced and contrast enhanced procedures hence doubling the ionizing radiation exposure. The purpose of this study was to determine the feasibility of single CT procedures in diagnosing PHNM.

METHODS

This was a cross-sectional study using CT images from patients, aged fifteen years and below, investigated for head and neck malignancies at the Uganda Cancer Institute. Three radiologists, observers A, B and C, with 12, 5 and 2 years of experience, respectively, participated in the study. They independently reported contrast enhanced images (protocol A), unenhanced images (protocol B), then both unenhanced and contrast enhanced images (protocol C) in 2 months intervals. Inter- and intra- observer agreement was determined using Gwen's Agreement coefficient.

RESULTS

Seventy-three CT scans of 36 boys and 37 girls, with a median age of 9 (3-13) years, were used. Intra-and inter-observer agreement on primary tumour location ranged from substantial to almost perfect with the highest intra-observer agreement observed when protocols A and C were compared. Inter-observer agreement for tumour calcifications was substantial for protocol A. Observers A and C demonstrated an almost perfect intra-observer agreement when protocols A and C were compared. There was a substantial inter-observer agreement on diagnosis for all protocols.

CONCLUSIONS

In our setting and examining a limited number of CT images, we demonstrated that contrast-enhanced CT scans provide sufficient information with no evidence of additional value of unenhanced images. Using contrast-enhanced images alone reduced the radiation exposure significantly.

Interobserver variability studies in diagnostic imaging: a methodological systematic review

OBJECTIVES

To review the methodology of interobserver variability studies; including current practice and quality of conducting and reporting studies.

METHODS

Interobserver variability studies between January 2019 and January 2020 were included; extracted data comprised of study characteristics, populations, variability measures, key results, and conclusions. Risk of bias was assessed using the COSMIN tool for assessing reliability and measurement error.

RESULTS

Seventy-nine full-text studies were included covering various imaging tests and clinical areas. The median number of patients was 47 (IQR:23-88), and observers were 4 (IQR:2-7), with sample size justified in 12 (15%) studies. Most studies used static images (n = 75, 95%), where all observers interpreted images for all patients (n = 67, 85%). Intraclass correlation coefficients (ICC) (n = 41, 52%), Kappa (κ) statistics (n = 31, 39%) and percentage agreement (n = 15, 19%) were most commonly used. Interpretation of variability estimates often did not correspond with study conclusions. The COSMIN risk of bias tool gave a very good/adequate rating for 52 studies (66%) including any studies that used variability measures listed in the tool. For studies using static images, some study design standards were not applicable and did not contribute to the overall rating.

CONCLUSIONS

Interobserver variability studies have diverse study designs and methods, the impact of which requires further evaluation. Sample size for patients and observers was often small without justification. Most studies report ICC and κ values, which did not always coincide with the study conclusion. High ratings were assigned to many studies using the COSMIN risk of bias tool, with certain standards scored 'not applicable' when static images were used.

ADVANCES IN KNOWLEDGE

The sample size for both patients and observers was often small without justification.For most studies, observers interpreted static images and did not evaluate the process of acquiring the imaging test, meaning it was not possible to assess many COSMIN risk of bias standards for studies with this design.Most studies reported intraclass correlation coefficient and κ statistics; study conclusions often did not correspond with results.

Prospective comparison of two methods for assessing sarcopenia and interobserver agreement on retrospective CT images

PURPOSE OF THE STUDY

To compare the relationships between two CT derived sarcopenia assessment methods, and compare their relationship with inter-rater and intrarater validations and colorectal surgical outcomes.

STUDY DESIGN

157 CT scans were identified across Leeds Teaching Hospitals National Health Service Trust for patients undergoing colorectal cancer surgery. 107 had body mass index data available, required to determine sarcopenia status. This work explores the relationship between sarcopenia, as measured by both total cross sectional-area (TCSA) and psoas-area (PA) and surgical outcomes. All images were assessed for inter-rater and intrarater variability for both TCSA and PA methods of sarcopenia identification. The raters included a radiologist, an anatomist and two medical students.

RESULTS

Prevalence of sarcopenia was different when measured by PA (12.2%-22.4%) in comparison to TCSA (60.8%-70.1%). Strong correlation exists between muscle areas in both TCSA and PA measures, however, there were significant differences between methods after the application of method-specific cut-offs. There was substantial agreement for both intrarater and inter-rater comparisons for both TCSA and PA sarcopenia measures. Outcome data were available for 99/107 patients. Both TCSA and PA have poor association with adverse outcomes following colorectal surgery.

CONCLUSIONS

CT-determined sarcopenia can be identified by junior clinicians, those with anatomical understanding and radiologists. Our study identified sarcopenia to have a poor association with adverse surgical outcomes in a colorectal population. Published methods of identifying sarcopenia are not translatable to all clinical populations. Currently available cut-offs require refinement for potential confounding factors, to provide more valuable clinical information.

WHAT IS ALREADY KNOWN ON THIS TOPIC?

WHAT DOES THIS STUDY ADD?

HOW MIGHT THIS STUDY AFFECT RESEARCH, PRACTICE OR POLICY?

Detection of grey zones in inter-rater agreement studies

BACKGROUND

In inter-rater agreement studies, the assessment behaviour of raters can be influenced by their experience, training levels, the degree of willingness to take risks, and the availability of clear guidelines for the assessment. When the assessment behaviour of raters differentiates for some levels of an ordinal classification, a grey zone occurs between the corresponding adjacent cells to these levels around the main diagonal of the table. A grey zone introduces a negative bias to the estimate of the agreement level between the raters. In that sense, it is crucial to detect the existence of a grey zone in an agreement table.

METHODS

In this study, a framework composed of a metric and the corresponding threshold is developed to identify grey zones in an agreement table. The symmetry model and Cohen's kappa are used to define the metric, and the threshold is based on a nonlinear regression model. A numerical study is conducted to assess the accuracy of the developed framework. Real data examples are provided to illustrate the use of the metric and the impact of identifying a grey zone.

RESULTS

The sensitivity and specificity of the proposed framework are shown to be very high under moderate, substantial, and near-perfect agreement levels for [Formula: see text] and [Formula: see text] tables and sample sizes greater than or equal to 100 and 50, respectively. Real data examples demonstrate that when a grey zone is detected in the table, it is possible to report a notably higher level of agreement in the studies.

CONCLUSIONS

The accuracy of the proposed framework is sufficiently high; hence, it provides practitioners with a precise way to detect the grey zones in agreement tables.

Inter- and intra-observer variability of computed tomography-based parenchymal- and ductal diameters in chronic pancreatitis: a multi-observer international study

PURPOSE

The need for incorporation of quantitative imaging biomarkers of pancreatic parenchymal and ductal structures has been highlighted in recent proposals for new scoring systems in chronic pancreatitis (CP). To quantify inter- and intra-observer variability in CT-based measurements of ductal- and gland diameters in CP patients.

MATERIALS AND METHODS

Prospectively acquired pancreatic CT examinations from 50 CP patients were reviewed by 12 radiologists and four pancreatologists from 10 institutions. Assessment entailed measuring maximum diameter in the axial plane of four structures: (1) pancreatic head (PDhead), (2) pancreatic body (PDbody), (3) main pancreatic duct in the pancreatic head (MPDhead), and (4) body (MPDbody). Agreement was assessed by the 95% limits of agreement with the mean (LOAM), representing how much a single measurement for a specific subject may plausibly deviate from the mean of all measurements on the specific subject. Bland-Altman limits of agreement (LoA) were generated for intra-observer pairs.

RESULTS

The 16 observers completed 6400 caliper placements comprising a first and second measurement session. The widest inter-observer LOAM was seen with PDhead (± 9.1 mm), followed by PDbody (± 5.1 mm), MPDhead (± 3.2 mm), and MPDbody (± 2.6 mm), whereas the mean intra-observer LoA width was ± 7.3, ± 5.1, ± 3.7, and ± 2.4 mm, respectively.

CONCLUSION

Substantial intra- and inter-observer variability was observed in pancreatic two-point measurements. This was especially pronounced for parenchymal and duct diameters of the pancreatic head. These findings challenge the implementation of two-point measurements as the foundation for quantitative imaging scoring systems in CP.

The cisterna chyli: a systematic review of definition, prevalence, and anatomy

The cisterna chyli is a lymphatic structure found at the caudal end of the thoracic duct that receives lymph draining from the abdominal and pelvic viscera and lower limbs. In addition to being an important landmark in retroperitoneal surgery, it is the key gateway for interventional radiology procedures targeting the thoracic duct. A detailed understanding of its anatomy is required to facilitate more accurate intervention, but an exhaustive summary is lacking. A systematic review was conducted, and 49 published human studies met the inclusion criteria. Studies included both healthy volunteers and patients and were not restricted by language or date. The detectability of the cisterna chyli is highly variable, ranging from 1.7 to 98%, depending on the study method and criteria used. Its anatomy is variable in terms of location (vertebral level of T10 to L3), size (ranging 2-32 mm in maximum diameter and 13-80 mm in maximum length), morphology, and tributaries. The size of the cisterna chyli increases in some disease states, though its utility as a marker of disease is uncertain. The anatomy of the cisterna chyli is highly variable, and it appears to increase in size in some disease states. The lack of well-defined criteria for the structure and the wide variation in reported detection rates prevent accurate estimation of its natural prevalence in humans.

A descriptive study of samples sizes used in agreement studies published in the PubMed repository

INTRODUCTION

A sample size justification is required for all studies and should give the minimum number of subjects to be recruited for the study to achieve its primary objective. The aim of this review is to describe sample sizes from agreement studies with continuous or categorical endpoints and different methods of assessing agreement, and to determine whether sample size justification was provided.

METHODS

Data were gathered from the PubMed repository with a time interval of 28th September 2018 to 28th September 2020. The search returned 5257 studies of which 82 studies were eligible for final assessment after duplicates and ineligible studies were excluded.

RESULTS

We observed a wide range of sample sizes. Forty-six studies (56%) used a continuous outcome measure, 28 (34%) used categorical and eight (10%) used both. Median sample sizes were 50 (IQR 25 to 100) for continuous endpoints and 119 (IQR 50 to 271) for categorical endpoints. Bland-Altman limits of agreement (median sample size 65; IQR 35 to 124) were the most common method of statistical analysis for continuous variables and Kappa coefficients for categorical variables (median sample size 71; IQR 50 to 233). Of the 82 studies assessed, only 27 (33%) gave justification for their sample size.

CONCLUSIONS

Despite the importance of a sample size justification, we found that two-thirds of agreement studies did not provide one. We recommend that all agreement studies provide rationale for their sample size even if they do not include a formal sample size calculation.

Interobserver Reliability and the Sigmoid Takeoff—An Interobserver Study

Simple Summary

Colorectal cancer is the second most common cancer. The sigmoid takeoff is the landmark where the colon sigmoid curves toward the sacrum viewed from sagittal magnetic resonance imaging. We assessed the interobserver variability in the assessment of the anal verge and anorectal junction in patients diagnosed with rectal cancer using magnetic resonance imaging. Our data indicate that radiologists are excellent at pointing out if a colorectal tumour is above or beneath the takeoff landmark.

Abstract

Background: Colorectal cancer is the second most common cancer worldwide. The sigmoid takeoff is the landmark where the colon sigmoid curves toward the sacrum viewed from sagittal magnetic resonance imaging (MRI). The purpose of this study was to assess interobserver variability in the assessment of the anal verge and anorectal junction in patients diagnosed with rectal cancer on magnetic resonance imaging (MRI). Materials and Methods: The rectal MRI examinations were performed using a 1.5- or 3.0-tesla unit using an anterior coil and a standard scan protocol. Two senior radiologists assessed MRI scans from patients under investigation for rectal cancer. The two observers assessed the anal verge and takeoff in cm independently. Difference in agreement between the observers were evaluated using intraclass correlation (ICC) and graphically by Bland–Altman plots. Results: The study population (n = 122) included 68 (55.7%) female and 54 (44.3%) male subjects. The overall median age was 69.5 years (range 39–95 years). There was perfect agreement between the two observers when defining rectal tumor above or below the takeoff landmark. The reliability of measuring the distance from the anal verge to the sigmoid takeoff was 0.712. Conclusion: Overall, the study found a moderate reliability in assessing the location of the sigmoid takeoff, with a low difference in the distance measuring, as well as a good consensus concerning the determination of tumors in relation to the sigmoid takeoff. Routine implementation of this information within the report seems reasonable.

Understanding the reliability of trial outcome measures: The example of angiographic results of surgical or endovascular treatments of aneurysms

BACKGROUND

The reliability of outcome measures is of central importance in clinical research. Studies of reliability remain rare in the neurovascular field.

METHODS

A narrative review of the history (1997-2021) of reporting angiographic results of the surgical or endovascular treatments of aneurysms serves to illustrate the importance of precisely defining outcome measures. We also review how the reliability of an angiographic classification system was studied.

DISCUSSION

Outcome measures are commonly used without precise definitions. When definitions or classification systems exist, they are rarely verified for their reliability. Twenty-five years following its introduction, a classification of angiographic results of aneurysm treatments is still being studied and modified.

CONCLUSION

The reliability of outcome measures should be made a research priority if we are to practice outcome-based medical or surgical care.

The challenge of applying digital image processing software on intraoral radiographs for osteoporosis risk assessment

OBJECTIVES

The purpose of this study was to evaluate rater agreement and the accuracy of a semi-automated software and its fully automated tool for osteoporosis risk assessment in intraoral radiographs.

METHODS

A total of 567 intraoral radiographs was selected retrospectively from women aged 75-80 years participating in a large population-based study (SUPERB) based in Gothenburg, Sweden. Five raters assessed participants' risk of osteoporosis in the intraoral radiographs using a semi-automated software. Assessments were repeated after 4 weeks on 121 radiographs (20%) randomly selected from the original 567. Radiographs were also assessed by the softwares' fully automated tool for analysis.

RESULTS

Overall interrater agreement for the five raters was 0.37 (95% CI 0.32-0.41), and for the five raters with the fully automated tool included as 'sixth rater' the overall Kappa was 0.34 (0.30-0.38). Intrarater agreement varied from moderate to substantial according to the Landis and Koch interpretation scale. Diagnostic accuracy was calculated in relation to reference standard for osteoporosis diagnosis which is T-score values for spine, total hip and femoral neck and presented in form of sensitivities, specificities, predictive values, likelihood ratios and odds ratios. All raters' mean sensitivity, including the fully automated tool, was 40,4% (range 14,3%-57,6%). Corresponding values for specificity was 69,5% (range 59,7%-90,4%). The diagnostic odds ratios ranged between 1 and 2.7.

CONCLUSION

The low diagnostic odds ratio and agreement between raters in osteoporosis risk assessment using the software for analysis of the trabecular pattern in intraoral radiographs shows that more work needs to be done to optimise the automation of trabecular pattern analysis in intraoral radiographs.

Agreement and differentiation of intradural spinal cord lesions in dogs using magnetic resonance imaging

Background

Magnetic resonance imaging is the method of choice for diagnosing spinal cord neoplasia, but the accuracy of designating the relationship of a neoplasm to the meninges and agreement among observers is unknown.

Objectives

To determine agreement among observers and accuracy of diagnosis compared with histology when diagnosing lesion location based on relationship to the meninges.

Animals

Magnetic resonance images from 53 dogs with intradural extramedullary and intramedullary spinal neoplasms and 17 dogs with degenerative myelopathy.

Methods

Six observers were supplied with 2 sets of 35 images at different time points and asked to designate lesion location. Agreement in each set was analyzed using kappa (κ) statistics. We tabulated total correct allocations and calculated sensitivity, specificity, and likelihood ratios for location designation from images compared with known histologic location for lesions confined to 1 location only.

Results

Agreement in the first set of images was moderate (κ = 0.51; 95% confidence interval [CI], 0.43‐0.58) and in the second, substantial (κ = 0.69; 95% CI, 0.66‐0.79). In the accuracy study, 180 (75%) of the 240 diagnostic calls were correct. Sensitivity and specificity were moderate to high for all compartments, except poor sensitivity was found for intradural extramedullary lesions. Positive likelihood ratios were high for intradural extramedullary lesions and degenerative myelopathy.

Conclusions and Clinical Importance

Overall accuracy in diagnosis was reasonable, and positive diagnostic calls for intradural extramedullary lesions and negative calls for intramedullary lesions are likely to be helpful. Observers exhibited considerable disagreement in designation of lesions relationship to the meninges.

Using MRI to quantify skeletal muscle pathology in Duchenne muscular dystrophy: A systematic mapping review

There is a great demand for accurate non‐invasive measures to better define the natural history of disease progression or treatment outcome in Duchenne muscular dystrophy (DMD) and to facilitate the inclusion of a large range of participants in DMD clinical trials. This review aims to investigate which MRI sequences and analysis methods have been used and to identify future needs. Medline, Embase, Scopus, Web of Science, Inspec, and Compendex databases were searched up to 2 November 2019, using keywords “magnetic resonance imaging” and “Duchenne muscular dystrophy.” The review showed the trend of using T1w and T2w MRI images for semi‐qualitative inspection of structural alterations of DMD muscle using a diversity of grading scales, with increasing use of T2map, Dixon, and MR spectroscopy (MRS). High‐field (>3T) MRI dominated the studies with animal models. The quantitative MRI techniques have allowed a more precise estimation of local or generalized disease severity. Longitudinal studies assessing the effect of an intervention have also become more prominent, in both clinical and animal model subjects. Quality assessment of the included longitudinal studies was performed using the Newcastle‐Ottawa Quality Assessment Scale adapted to comprise bias in selection, comparability, exposure, and outcome. Additional large clinical trials are needed to consolidate research using MRI as a biomarker in DMD and to validate findings against established gold standards. This future work should use a multiparametric and quantitative MRI acquisition protocol, assess the repeatability of measurements, and correlate findings to histologic parameters.

A rater agreement study on measurements in cross-sectional CBCT images exploring the association between alveolar bone morphology and craniofacial height

Objectives

To investigate rater agreement regarding measurements of height and width of the maxilla and mandible using cross-sectional images from CBCT examinations. Furthermore, to explore the association between vertical craniofacial height and alveolar bone morphology.

Methods

Pre-treatment CBCT scans from 450 patients referred for treatment to a private clinic for orthodontics and oral surgery in Scandinavia were available and of these, 180 were selected. Lateral head images were generated from the CBCT volumes to categorise subjects into three groups based on their craniofacial height. Cross-sectional images of the maxillary and mandibular bodies at three locations in the maxilla and mandible, respectively, were obtained and measured at one height and two width recordings by five raters. One-way analysis of variance with a Tukey post hoc test was performed. A significance level of 5% was used.

Results

Rater agreement was mostly excellent or good when measuring height and width of the maxilla and mandible in cross-sectional CBCT images. For height (of the alveolar bone/bodies), there were statistically significant differences between the low- and the high-angle groups for all the observers when measuring in the premolar and midline regions, both in the maxilla and in the mandible.

Conclusion

The high agreement found ensures a reliable measurement technique and confirms the relation between craniofacial height and alveolar bone height and width.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11282-020-00493-4.

Measuring clinical uncertainty and equipoise by applying the agreement study methodology to patient management decisions

Background

Clinical uncertainty and equipoise are vague notions that play important roles in contemporary problems of medical care and research, including the design and conduct of pragmatic trials. Our goal was to show how the reliability study methods normally used to assess diagnostic tests can be applied to particular management decisions to measure the degree of uncertainty and equipoise regarding the use of rival management options.

Methods

We first use thrombectomy in acute stroke as an illustrative example of the method we propose. We then review, item by item, how the various design elements of diagnostic reliability studies can be modified in order to measure clinical uncertainty.

Results

The thrombectomy example shows sufficient disagreement and uncertainty to warrant the conduct of additional randomized trials. The general method we propose is that a sufficient number of diverse individual cases sharing a similar clinical problem and covering a wide spectrum of clinical presentations be assembled into a portfolio that is submitted to a variety of clinicians who routinely manage patients with the clinical problem.

Discussion

Clinicians are asked to independently choose one of the predefined management options, which are selected from those that would be compared within a randomized trial that would address the clinical dilemma. Intra-rater agreement can be assessed at a later time with a second evaluation. Various professional judgments concerning individual patients can then be compared and analyzed using kappa statistics or similar methods. Interpretation of results can be facilitated by providing examples or by translating the results into clinically meaningful summary sentences.

Conclusions

Measuring the uncertainty regarding management options for clinical problems may reveal substantial disagreement, provide an empirical foundation for the notion of equipoise, and inform or facilitate the design/conduct of clinical trials to address the clinical dilemma.

Morphometric Optical Imaging of Microporated Nail Tissue: An Investigation of Intermethod Agreement, Reliability, and Technical Limitations

BACKGROUND AND OBJECTIVES

While optical imaging is a useful technique to quantitate morphological differences and treatment effects, comparative investigations of the various techniques are lacking. This study aimed at evaluating intermethod agreement, reliability, and technical limitations of wide-field microscopy (WFM), reflectance confocal microscopy (RCM), and optical coherence tomography (OCT) for morphometry by assessing fractionally ablated nail tissue.

STUDY DESIGN/MATERIALS AND METHODS

Fifty healthy nail clippings were processed with a fractionated CO₂ -laser (20 mJ/microbeam, density 15%), measured with calipers, and imaged using WFM, OCT, and RCM. Images were assessed for nail plate thickness, micropore dimensions, degree of poration, and artifacts. Repeated measurements (2-5) were taken to evaluate method repeatability using Cronbach's α and coefficients of variation (CoV), and estimate the intermethod correlation through linear correlation assessment (Pearson correlation coefficient [PCC]), ranked correlation (Kendall's tau; tau-c), and intraclass correlation (Shrout-Fleiss reliability coefficient; ICC).

RESULTS

The repeatability varied substantially between methods and target measurements. The level of intermethod agreement for thickness measurements performed with calipers, WFM, and OCT was high (tau-c ≥ 0.7; ICC ≥ 0.8; PCC ≥ 0.9). RCM could only image 28 out of 50 samples due to its limited penetration depth. OCT demonstrated the highest repeatability of all imaging techniques (CoV 4-7%) and nail thickness showed the highest measurement reliability (α = 0.92). Micropore dimensions correlated strongest between OCT and RCM (tau-c/ICC/PCC ≥ 0.5). All modalities were prone to artifacts, which may have adversely affected measurement variation and intermethod agreement.

CONCLUSION

Intermethod agreement and reliability appear to be highly dependent on the specific modality and target measurement. To reap the benefits of each technique while mitigating their limitations, an integrated approach to optical imaging is recommended. Lasers Surg. Med. © 2020 Wiley Periodicals LLC.

Reliability of CT Angiography in Cerebral Vasospasm: A Systematic Review of the Literature and an Inter- and Intraobserver Study

BACKGROUND AND PURPOSE

Computed tomography angiography offers a non-invasive alternative to DSA for the assessment of cerebral vasospasm following subarachnoid hemorrhage but there is limited evidence regarding its reliability. Our aim was to perform a systematic review (Part I) and to assess (Part II) the inter- and intraobserver reliability of CTA in the diagnosis of cerebral vasospasm.

MATERIALS AND METHODS

In Part I, articles reporting the reliability of CTA up to May 2018 were systematically searched and evaluated. In Part II, 11 raters independently graded 17 arterial segments in each of 50 patients with SAH for the presence of vasospasm using a 4-category scale. Raters were additionally asked to judge the presence of any moderate/severe vasospasm (≥ 50% narrowing) and whether findings would justify augmentation of medical treatment or conventional angiography ± balloon angioplasty. Four raters took part in the intraobserver reliability study.

RESULTS

In Part I, the systematic review revealed few studies with heterogeneous vasospasm definitions. In Part II, we found interrater reliability to be moderate at best (κ ≤ 0.6), even when results were stratified according to specialty and experience. Intrarater reliability was substantial (κ > 0.6) in 3/4 readers. In the per arterial segment analysis, substantial agreement was reached only for the middle cerebral arteries, and only when senior raters' judgments were dichotomized (presence or absence of ≥50% narrowing). Agreement on the medical or angiographic management of vasospasm based on CTA alone was less than substantial (κ ≤ 0.6).

CONCLUSIONS

The diagnosis of vasospasm using CTA alone was not sufficiently repeatable among observers to support its general use to guide decisions in the clinical management of patients with SAH.

How reliable is the lacrimal scintigraphy report? An inter-observer agreement and reliability study

AIM

To assess the inter-observer agreement and reliability as well as intra-observer repeatability for lacrimal scintigraphy (LS) reports with and without considering the irrigation test results.

METHODS

A prospective, observational, cross sectional study. Two masked clinicians (lacrimal surgeon and nuclear medicine specialist) independently reported 100 LS images (50 patients of >6 years of age with unilateral anophthalmic socket) in a university hospital. The lacrimal surgeon performed a diagnostic irrigation test and repeated the report of the same LS images 2y after the first report (intra-observer agreement). A weighted Kappa analysis was performed to determine inter-observer agreement and reliability as well as intra-observer repeatability for the type (normal, partial and complete obstruction) and location (presac, preduct, and intraduct) of the obstruction. Subgroup analysis was also performed with consideration of irrigation test results.

RESULTS

A significantly moderate agreement was found between lacrimal surgeon and nuclear medicine specialist for both the type (Kappa=0.55) and location (Kappa=0.48) of obstruction. Agreement values were higher for the type (Kappa=0.61 vs 0.41) and location (Kappa=0.56 vs 0.31) of obstruction in cases with normal than abnormal irrigation test. Strong and significant intra-observer (lacrimal surgeon) repeatability was found for both the type (Kappa=0.66) and location (Kappa=0.69) of obstruction. LS showed no to slight reliability based on irrigation test.

CONCLUSION

A moderate agreement is found between lacrimal surgeon and nuclear medicine specialist regarding the interpretation of LS suggesting the importance of consensus groups among nuclear medicine specialists and lacrimal surgeons to create a common language for interpretation of LS. Intra-observer repeatability is strong for the lacrimal surgeon.

Hemorrhagic transformation after stroke: inter- and intrarater agreement

BACKGROUND AND PURPOSE

Hemorrhagic transformation (HT) is a complication of stroke that can occur spontaneously or after treatment. We aimed to assess the inter- and intrarater reliability of HT diagnosis.

METHODS

Studies assessing the reliability of the European Cooperative Acute Stroke Study (ECASS) classification of HT or of the presence (yes/no) of HT were systematically reviewed. A total of 18 raters independently examined 30 post-thrombectomy computed tomography scans selected from the Aspiration versus STEnt-Retriever (ASTER) trial. They were asked whether there was HT (yes/no), what the ECASS classification of the particular scan (0/HI1/HI2/PH1/PH2) (HI indicates hemorrhagic infarctions and PH indicates parenchymal hematomas) was and whether they would prescribe an antiplatelet agent if it was otherwise indicated. Agreement was measured with Fleiss' and Cohen's κ statistics.

RESULTS

The systematic review yielded four studies involving few (≤3) raters with heterogeneous results. In our 18-rater study, agreement for the presence of HT was moderate [κ = 0.55; 95% confidence interval (CI), 0.41-0.68]. Agreement for ECASS classification was only fair for all five categories, but agreement improved to substantial (κ = 0.72; 95% CI, 0.69-0.75) after dichotomizing the ECASS classification into 0/HI1/HI2/PH1 versus PH2. The inter-rater agreement for the decision to reintroduce antiplatelet therapy was moderate for all raters, but substantial among vascular neurologists (κ = 0.70; 95% CI, 0.57-0.84).

CONCLUSION

The ECASS classification may involve too many categories and the diagnosis of HT may not be easily replicable, except in the presence of a large parenchymal hematoma.

Inter- and intraobserver reliability for angiographic leptomeningeal collateral flow assessment by the American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) scale

BACKGROUND

The adequacy of leptomeningeal collateral flow has a pivotal role in determining clinical outcome in acute ischemic stroke. The American Society of Interventional and Therapeutic Neuroradiology/Society of Interventional Radiology (ASITN/SIR) collateral score is among the most commonly used scales for measuring this flow. It is based on the extent and rate of retrograde collateral flow to the impaired territory on angiography.

OBJECTIVE

To evaluate inter- and intraobserver agreementin angiographic leptomeningeal collateral flow assessment.

MATERIALS AND METHODS

Thirty pretreatment angiogram video loops (frontal and lateral view), chosen from the randomized controlled trial THRombectomie des Artères CErebrales (THRACE), were sent for grading in an electronic file. 19 readers participated, including eight who had access to a training set before the first grading. 13 readers made a double evaluation, 3 months apart.

RESULTS

Overall agreement among the 19 observers was poor (κ = 0,16 ± 6,5.10 ^-3), and not improved with prior training (κ = 0,14 ± 0,016). Grade 4 showed the poorest interobserver agreement (κ=0.18±0.002) while grades 0 and 1 were associated with the best results (κ=0.52±0.001 and κ=0.43±0.004, respectively). Interobserver agreement increased (κ = 0,27± 0,014) when a dichotomized score, 'poor collaterals' (score of 0, 1 or 2) versus 'good collaterals' (score of 3 or 4) was used. The intraobserver agreements varied between slight (κ=0.18±0.13) and substantial (κ=0.74±0.1), and were slightly improved with the dichotomized score (from κ=0.19±0.2 to κ=0.79±0.11).

CONCLUSION

Inter- and intraobserver agreement of collateral circulation grading using the ASITN/SIR score was poor, raising concerns about comparisons among publications. A simplified dichotomized judgment may be a more reproducible assessment when images are rated by the same observer(s) in randomized trials.