How to assess intra- and inter-observer agreement with quantitative PET using variance component analysis: a proposal for standardisation

Optimizing SUV Analysis: A Multicenter Study on Preclinical FDG-PET/CT Highlights the Impact of Standardization

PURPOSE

Preclinical imaging, with translational potential, lacks a standardized method for defining volumes of interest (VOIs), impacting data reproducibility. The aim of this study was to determine the interobserver variability of VOI sizes and standard uptake values (SUVmean and SUVmax) of different organs using the same [18F]FDG-PET and PET/CT datasets analyzed by multiple observers. In addition, the effect of a standardized analysis approach was evaluated.

PROCEDURES

In total, 12 observers (4 beginners and 8 experts) analyzed identical preclinical [18F]FDG-PET-only and PET/CT datasets according to their local default image analysis protocols for multiple organs. Furthermore, a standardized protocol was defined, including detailed information on the respective VOI size and position for multiple organs, and all observers reanalyzed the PET/CT datasets following this protocol.

RESULTS

Without standardization, significant differences in the SUVmean and SUVmax were found among the observers. Coregistering CT images with PET images improved the comparability to a limited extent. The introduction of a standardized protocol that details the VOI size and position for multiple organs reduced interobserver variability and enhanced comparability.

CONCLUSIONS

The protocol offered clear guidelines and was particularly beneficial for beginners, resulting in improved comparability of SUVmean and SUVmax values for various organs. The study suggested that incorporating an additional VOI template could further enhance the comparability of the findings in preclinical imaging analyses.

Impact of the evaluation method on 4D flow-derived diastolic transmitral and myocardial peak velocities: Comparison with echocardiography

PURPOSE

To compare agreement of different evaluation methods of magnetic resonance (MR) 4D flow-derived diastolic transmitral and myocardial peak velocities as well as their ratios, using echocardiography as reference.

METHODS

In this prospective study, 60 subjects without symptoms of cardiovascular disease underwent echocardiography and non-contrast 3 T MR 4D flow imaging of the heart. Early- (E) and late-diastolic (A) transmitral peak filling velocities were evaluated from 4D flow data using three different strategies: 1) at the mitral valve tips in short-axis orientation (SA-method), 2) between the mitral valve tips in 4-chamber orientation (4-chamber-method), and 3) as maximal velocities in the transmitral inflow volume (max-velocity-method). Septal, lateral and average early-diastolic myocardial peak velocities (e') were derived from the myocardial tissue in the vicinity of the mitral valve. 4D flow parameters were compared with echocardiography by correlation and Bland-Altman analysis.

RESULTS

All 4D flow-derived E, A and E/A values correlated with echocardiography (r = 0.65-0.73, 0.75-0.83 and 0.74-0.86, respectively). While the SA- and 4-chamber-methods substantially underestimated E and A compared to echocardiography (p < 0.001), the max-velocity-method provided E (p = 0.13) and E/A (p = 0.07) without significant bias. Septal, lateral and average e' from 4D flow as well as the max-velocity-method-derived E/e' correlated with echocardiographic measurements (r = 0.64-0.81) and showed no significant bias (p = 0.26-0.54).

CONCLUSION

MR 4D flow imaging allows precise and accurate evaluation of transmitral and myocardial peak velocities for characterization of LV diastolic function without significant bias to echocardiography, when transmitral velocities are assessed from the transmitral inflow volume. This enables the use of validated echocardiography threshold values.

Clinical assessment of breast symmetry and aesthetic outcome: can 3D imaging be the gold standard?

There is a lack of an accurate standardised objective method to assess aesthetic outcome after breast surgery. In this methodological study, we investigated the intra- and inter-observer reproducibility of breast symmetry and volume assessed using three-dimensional surface imaging (3D-SI), evaluated the reproducibility depending on imaging posture, and proposed a new combined volume-shape-symmetry (VSS) parameter. Images were acquired using the VECTRA XT 3D imaging system, and analysed by two observers using VECTRA Analysis Module. Breast symmetry was measured through the root mean square distance. All women had undergone bilateral risk-reducing mastectomy and immediate breast reconstruction. The reproducibility and correlations of breast symmetry and volume measurements were compared using Bland-Altman's plots and tested with Spearman's rank correlation coefficient. 3D surface images of 58 women were analysed (348 symmetry measurements, 696 volume measurements). The intra-observer reproducibility of breast symmetry measurements was substantial-excellent, the inter-observer reproducibility was substantial, and the inter-posture reproducibility was substantial. For measurements of breast volumes, the intra-observer reproducibility was excellent, the inter-observer reproducibility was moderate-substantial, and the inter-posture reproducibility was substantial-excellent. The intra-observer reproducibility of VSS was excellent while the inter-observer reproducibility was substantial for both observers, independent of posture. There were no statistically strong correlations between breast symmetry and volume differences. The intra-observer reproducibility was found to be substantial-excellent for several 3D-SI measurements independent of imaging posture. However, the inter-observer reproducibility was lower than the intra-observer reproducibility, indicating that 3D-SI in its present form is not a great assessment for symmetry.

A Deep Learning Algorithm for Radiographic Measurements of the Hip in Adults-A Reliability and Agreement Study

Hip dysplasia (HD) is a frequent cause of hip pain in skeletally mature patients and may lead to osteoarthritis (OA). An accurate and early diagnosis may postpone, reduce or even prevent the onset of OA and ultimately hip arthroplasty at a young age. The overall aim of this study was to assess the reliability of an algorithm, designed to read pelvic anterior-posterior (AP) radiographs and to estimate the agreement between the algorithm and human readers for measuring (i) lateral center edge angle of Wiberg (LCEA) and (ii) Acetabular index angle (AIA). The algorithm was based on deep-learning models developed using a modified U-net architecture and ResNet 34. The newly developed algorithm was found to be highly reliable when identifying the anatomical landmarks used for measuring LCEA and AIA in pelvic radiographs, thus offering highly consistent measurement outputs. The study showed that manual identification of the same landmarks made by five specialist readers were subject to variance and the level of agreement between the algorithm and human readers was consequently poor with mean measured differences from 0.37 to 9.56° for right LCEA measurements. The algorithm displayed the highest agreement with the senior orthopedic surgeon. With further development, the algorithm may be a good alternative to humans when screening for HD.

Sample size determination in method comparison and observer variability studies

The comparison of two quantitative measuring devices is often performed with the Limits of Agreement proposed by Bland and Altman in their seminal Lancet paper back in 1986. Sample size considerations were rare for such agreement analyses in the past, but recently several proposals have been made depending on how agreement is to be assessed and the number of replicates to be used. We have summarized recent developments and recommendations in various situations including a distinction between method comparison and observer variability studies. These include current state-of-the-art analysis of and reporting guidelines for agreement studies. General recommendations close the paper.

Morphometric study of the ventricular indexes in healthy ovine BRAIN using MRI

Background

Sheep (Ovis aries) have been largely used as animal models in a multitude of specialties in biomedical research. The similarity to human brain anatomy in terms of brain size, skull features, and gyrification index, gives to ovine as a large animal model a better translational value than small animal models in neuroscience. Despite this evidence and the availability of advanced imaging techniques, morphometric brain studies are lacking. We herein present the morphometric ovine brain indexes and anatomical measures developed by two observers in a double-blinded study and validated via an intra- and inter-observer analysis.

Results

For this retrospective study, T1-weighted Magnetic Resonance Imaging (MRI) scans were performed at 1.5 T on 15 sheep, under general anaesthesia. The animals were female Ovis aries, in the age of 18-24 months. Two observers assessed the scans, twice time each. The statistical analysis of intra-observer and inter-observer agreement was obtained via the Bland-Altman plot and Spearman rank correlation test.

The results are as follows (mean ± Standard deviation):

Indexes: Bifrontal 0,338 ± 0,032 cm; Bicaudate 0,080 ± 0,012 cm; Evans’ 0,218 ± 0,035 cm; Ventricular 0,241 ± 0,039 cm; Huckman 1693 ± 0,174 cm; Cella Media 0,096 ± 0,037 cm; Third ventricle ratio 0,040 ± 0,007 cm.

Anatomical measures: Fourth ventricle length 0,295 ± 0,073 cm; Fourth ventricle width 0,344 ± 0,074 cm; Left lateral ventricle 4175 ± 0,275 cm; Right lateral ventricle 4182 ± 0,269 cm; Frontal horn length 1795 ± 0,303 cm; Interventricular foramen left 1794 ± 0,301 cm; Interventricular foramen right 1,78 ± 0,317 cm.

Conclusions

The present study provides baseline values of linear indexes of the ventricles in the ovine models. The acquisition of these data contributes to filling the knowledge void on important anatomical and morphological features of the sheep brain.

Variability of echocardiographic measures of left ventricular diastolic function. The HUNT study

OBJECTIVE

Investigate variability related to image acquisition and reading process for echocardiographic measures of left ventricular (LV) diastolic function, and its influence on classification of LV diastolic dysfunction (LVDD).

METHODS

Forty participants (19 women) mean age 62 (28-88) years underwent echocardiographic examinations twice by different echocardiographers and blinded analyses by four readers in a cross-sectional design. Measurements included quantification of two- (2D) and three-dimensional (3D) recordings of the left atrium (LA) (maximal) volume (LAV_max ) and spectral Doppler blood flow and tissue velocities for assessment of LV diastolic function. Variability and reproducibility measures were calculated using variance component analyses and Kappa statistics.

RESULTS

Image acquisition influenced variability more than image reading (mean 24% and 4% of variance, respectively), but variability from image reading was especially important for 2D LAV_max (16% of variance) compared to 4% for 3D LAV_max , which was reflected in better agreement for 3D measures. The variability of measures used in classification of LVDD had clinical significance, and agreement across the four raters in classification using current recommendations was only fair (Kappa 0.42), but the agreement improved when using 3D LAV_max (Kappa 0.58). Agreement and reliability measures were reported for all measures.

CONCLUSION

Performing a new image acquisition influenced variability more than a introducing a new image reader, but there were differences across the different measures. LAV_max by 3D is superior to 2D with respect to lower variability. The variability of diastolic measures influences the reliability of LVDD classification, and this should be taken into account in the everyday clinic.

Measures of diagnostic precision (repeatability and reproducibility) for three test methods designed to detect spring viremia of carp virus

Spring viremia of carp virus (SVCV) is a rhabdovirus of the Sprivivirus genus and the etiological agent of an internationally regulated aquatic animal disease in several fish species, including koi carp Cyprinus carpio L. The virus has a complex lifecycle with both acute and persistent stages of infection and can cause high mortality in affected populations. In this study, the diagnostic repeatability (within laboratory agreement) and reproducibility (between laboratory agreement) of 3 tests were investigated to assess their fitness as SVCV diagnostic tools. The tests, reverse transcription quantitative polymerase chain reaction (RT-qPCR) assays targeting either the SVCV glycoprotein (Q1G) or nucleoprotein (Q2N) genes and virus isolation by cell culture (VI), were performed in a blinded study with four Canadian laboratories. Test panels consisted of duplicate sets of 100 tissue samples collected from 3 SVCV prevalence populations of koi: a low-prevalence negative reference population (n = 20 fish) as well as moderate- (n = 50 fish) and high-prevalence (n = 30 fish) populations of koi experimentally infected with SVCV. The Q1G and Q2N tests were performed with kidney tissue in 3 laboratories and with brain tissue in 1 laboratory whereas pools of kidney, spleen and gill tissues were tested with the VI assay in 2 laboratories. Agreement of binary results was evaluated using the observed proportion of agreement, Cohen's kappa and Gwet's agreement coefficient (AC1) whereas the concordance correlation coefficient (ccc) and Bland Altman's limit of agreement were used to evaluate agreement of the RT-qPCR continuous data. Gwet's AC1 provided a more stable estimate of agreement than Cohen's kappa. Overall, high repeatability (AC1, 0.78-0.90) and reproducibility (AC1, 0.74-0.89) were observed for the Q1G and Q2N tests when kidney tissue was used. Lower agreement estimates of repeatability (AC1, 0.54-0.77) and reproducibility (AC1, 0.50-0.80) were obtained for the VI test. RT-qPCR reproducibility was low with kidney-brain tissue pairs (AC1, 0.09-0.46) and high with inter-test pairs of brain (AC1, 0.76-0.86) or kidney tissue (0.75-0.86). Tissue-specific differences in virus load affected test precision and informed final tissue selection. Repeatability (ccc, 0.94-0.97) and reproducibility (ccc, 0.91-0.97) estimates of agreement for paired continuous data from the RT-qPCR assays were similarly high with kidney tissue and lower with paired brain (ccc, 0.15-0.83) and kidney-brain tissues (ccc, 0.01-0.55). The high precision of Q1G and Q2N with kidney tissue suggests that the tests are performing similarly and are suitable candidates for assessment of their diagnostic accuracy.

Evaluating Correlations of Coracohumeral Ligament Thickness with Restricted Shoulder Range of Motion and Clinical Duration of Adhesive Capsulitis with Ultrasound Measurements

OBJECTIVES

The primary objective of this study is to evaluate, using ultrasound measurements, the correlation between coracohumeral ligament (CHL) thickness and restricted shoulder range of motion (ROM) in patients with adhesive capsulitis (AC). The secondary objective is to investigate the correlation between CHL thickness and disease duration.

DESIGN

Prospective cross-sectional survey.

SETTING

Clinical research of a tertiary care hospital.

METHODS

Overall, 65 patients with clinically diagnosed AC were enrolled. Ultrasound measurements of CHL thickness in the axial oblique plane were obtained under maximal external rotation of the glenohumeral joint. Both Shoulder Pain and Disability Index (SPADI) and shoulder ROM were prospectively evaluated by an experienced investigator. CHL thickness was compared with shoulder ROM and SPADI. The association between CHL thickness and disease duration was also investigated.

RESULTS

Simple linear regression analysis showed significant inverse correlation between CHL thickness and shoulder ROM including external rotation (ER) (r = -0.335, P = .006) and internal rotation (IR) (r = -0.409, P = .001). CHL thickness also correlated with disease duration (r = -0.352, P = .004). Multiple linear regression analysis demonstrated that CHL thickness was significantly associated with restricted ER (r = -0.293, P = .02) and IR (r = -0.363, P = .003) after adjusting for age and disease duration. On the other hand, CHL thickness showed no significant correlation with abduction (r = -0.210, P = .09), flexion (r = -0.170, P = .176), or total SPADI score (r = 0.176, P = .16). Moreover, CHL was significantly thicker in patients with disease duration >6 months (P = .004, difference in means: 0.55 mm, 95% confidence interval: -0.922, -0.183).

CONCLUSIONS

CHL was significantly thicker in later-stage AC. CHL thickness correlated negatively with ER and IR of the shoulder. Furthermore, CHL thickening could be observed in the early stage of the disease course. These imaging findings may assist in confirming the diagnosis of AC, leading to early intervention and treatment options.

Reporting Standards for a Bland-Altman Agreement Analysis: A Review of Methodological Reviews

The Bland-Altman Limits of Agreement is a popular and widespread means of analyzing the agreement of two methods, instruments, or raters in quantitative outcomes. An agreement analysis could be reported as a stand-alone research article but it is more often conducted as a minor quality assurance project in a subgroup of patients, as a part of a larger diagnostic accuracy study, clinical trial, or epidemiological survey. Consequently, such an analysis is often limited to brief descriptions in the main report. Therefore, in several medical fields, it has been recommended to report specific items related to the Bland-Altman analysis. The present study aimed to identify the most comprehensive and appropriate list of items for such an analysis. Seven proposals were identified from a MEDLINE/PubMed search, three of which were derived by reviewing anesthesia journals. Broad consensus was seen for the a priori establishment of acceptability benchmarks, estimation of repeatability of measurements, description of the data structure, visual assessment of the normality and homogeneity assumption, and plotting and numerically reporting both bias and the Bland-Altman Limits of Agreement, including respective 95% confidence intervals. Abu-Arafeh et al. provided the most comprehensive and prudent list, identifying 13 key items for reporting (Br. J. Anaesth. 2016, 117, 569-575). An exemplification with interrater data from a local study accentuated the straightforwardness of transparent reporting of the Bland-Altman analysis. The 13 key items should be applied by researchers, journal editors, and reviewers in the future, to increase the quality of reporting Bland-Altman agreement analyses.

Adaptive trial designs in diagnostic accuracy research

The aim of diagnostic accuracy studies is to evaluate how accurately a diagnostic test can distinguish diseased from nondiseased individuals. Depending on the research question, different study designs and accuracy measures are appropriate. As the prior knowledge in the planning phase is often very limited, modifications of design aspects such as the sample size during the ongoing trial could increase the efficiency of diagnostic trials. In intervention studies, group sequential and adaptive designs are well established. Such designs are characterized by preplanned interim analyses, giving the opportunity to stop early for efficacy or futility or to modify elements of the study design. In contrast, in diagnostic accuracy studies, such flexible designs are less common, even if they are as important as for intervention studies. However, diagnostic accuracy studies have specific features, which may require adaptations of the statistical methods or may lead to specific advantages or limitations of sequential and adaptive designs. In this article, we summarize the current status of methodological research and applications of flexible designs in diagnostic accuracy research. Furthermore, we indicate and advocate future development of adaptive design methodology and their use in diagnostic accuracy trials from an interdisciplinary viewpoint. The term "interdisciplinary viewpoint" describes the collaboration of experts of the academic and nonacademic research.

Intra- and inter-laboratory agreement of the disc diffusion assay for assessing antimicrobial susceptibility of porcine Escherichia coli

Reliable assessment of the susceptibility of animal bacterial pathogens to antimicrobials is of paramount importance in the fight against antimicrobial resistance. This work aims to estimate the repeatability (intra-laboratory agreement) and reproducibility (inter-laboratory agreement) of the disc diffusion assay in veterinary laboratories to understand further if the assay has a role in the surveillance of antimicrobial resistance in animals. Seven major veterinary laboratories from all States in Australia participated, and each tested the same panel of isolates five times at three to four-week intervals, against six antimicrobial agents using Clinical and Laboratory Standards Institute protocols. The panel consisted of twenty different isolates from porcine Escherichia coli from clinical cases and a single reference strain (ATCC 25922). Laboratories were blinded to the identity of the isolates, replicates, and to each other. In total, 4200 inhibition zone diameters (mm) were collected, and analysed descriptively, graphically, and with linear mixed models. Regardless of the laboratories and isolate/antimicrobial combinations, the overall very major error rate (proportion of isolates classified as susceptible when actual status is resistant) was 1.6%; the major error rate (proportion of isolates classified as resistant when actual status is susceptible) was 1.6%; and the 'minor error' rate (proportion of isolates with intermediate susceptibility that measure fully susceptible or resistant or vice versa) was 2.4%. The variation between repeated measurements ranged between 4.4-7.2 mm depending on the antimicrobial agent assessed. The reproducibility was always more variable than the repeatability, which suggested some laboratory effects. The repeatability coefficient of disc diffusion was lowest for tetracycline (4.4 mm, 95% CI: 3.8-5.0 mm) and ampicillin (4.6 mm, 95% CI: 4.2-5.2 mm) and highest for trimethoprim-sulfamethoxazole (6.6 mm, 95% CI: 5.9-7.4 mm). The reproducibility coefficient of disc diffusion was lowest for gentamicin (5.4, 95% CI: 4.0-7.2) and highest for trimethoprim-sulfamethoxazole (7.2 mm, 95%CI: 4.5-11.7 mm). The precision of the disc diffusion assay was deemed satisfactory for use in a national surveillance program for clinical porcine E. coli isolates. However, measurement variation of the disc diffusion assay is of concern for isolates with marginal susceptibility or resistance due to increased risk of misclassification.

Experience Applying the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) Indicated Five Questions Should Be Addressed in the Planning Phase from a Statistical Point of View

The Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed in 2011 to support transparent and accurate reporting. These studies may be conducted with the primary aim of estimating reliability and/or agreement itself, but are more often than not part of larger diagnostic accuracy studies, clinical trials, or epidemiological studies. As such, the study design may be compromised in terms of practicability issues, preventing the collection of sufficient results. We presented an example from a consultancy with a difficult mission and discussed five questions that concern the very nature of such a study (agreement vs. reliability; intra- vs. interrater), the rater population, explanatory factors in a multivariable model, and the statistical analysis strategy. Discussion of such basic methodological and statistical questions must take place before an investigation is started in order to ensure adequate data collection, to predict possible complications in the study, to plan sufficient statistical analyses, and to request timely assistance from an experienced statistician. GRRAS and its accompanying checklist of 15 items proved to be most helpful. Hopefully, our commentary will help improve the planning of agreement and reliability studies, which, in turn, will then be more focused, more appropriate, and more easily reported using GRRAS.

Group-sequential analysis may allow for early trial termination: illustration by an intra-observer repeatability study

Background

Group-sequential testing is widely used in pivotal therapeutic, but rarely in diagnostic research, although it may save studies, time, and costs. The purpose of this paper was to demonstrate a group-sequential analysis strategy in an intra-observer study on quantitative FDG-PET/CT measurements, illuminating the possibility of early trial termination which implicates significant potential time and resource savings.

Methods

Primary lesion maximum standardised uptake value (SUVmax) was determined twice from preoperative FDG-PET/CTs in 45 ovarian cancer patients. Differences in SUVmax were assumed to be normally distributed, and sequential one-sided hypothesis tests on the population standard deviation of the differences against a hypothesised value of 1.5 were performed, employing an alpha spending function. The fixed-sample analysis (N = 45) was compared with the group-sequential analysis strategies comprising one (at N = 23), two (at N = 15, 30), or three interim analyses (at N = 11, 23, 34), respectively, which were defined post hoc.

Results

When performing interim analyses with one third and two thirds of patients, sufficient agreement could be concluded after the first interim analysis and the final analysis. Other partitions did not suggest early stopping after adjustment for multiple testing due to one influential outlier and our small sample size.

Conclusions

Group-sequential testing may enable early stopping of a trial, allowing for potential time and resource savings. The testing strategy must, though, be defined at the planning stage, and sample sizes must be reasonably large at interim analysis to ensure robustness against single outliers. Group-sequential testing may have a place in accuracy and agreement studies.

Electronic supplementary material

The online version of this article (10.1186/s13550-017-0328-6) contains supplementary material, which is available to authorized users.

Intra- and inter-observer agreement and reliability of bone mineral density measurements around acetabular cup: a porcine ex-vivo study using single- and dual-energy computed tomography

Background

Periprosthetic bone loss is considered to be a potentially contributing factor in aseptic loosening of acetabular hip components, but no studies have shown this association. The lack of association might be caused by insufficient image quality because of metal artifacts and challenges in measuring bone density (BMD) in complex anatomic structures which might be overcome using dual-energy computed tomography (DECT).

Purpose

To test inter- and intra-observer agreement and reliability of in-house segmentation software measuring BMD adjacent to acetabular cup and to compare measurements performed with single-energy CT (SECT) and DECT in cemented and cementless cups.

Material and Methods

Twenty-four acetabular cups inserted in porcine hip specimens were scanned with SECT and DECT. Bone density was measured in a three-dimensional volume adjacent to the cup. Double measurements were performed.

Results

BMD derived from SECT was approximately four times higher than that of DECT. In both scan modes, intraclass correlation coefficient (ICC) was >0.90 with no differences between repeated measurements, except for uncemented cups where a statistically significant difference of 11 mg/cm³ was found with DECT. DECT showed narrower limits of agreement than SECT. Inter-observer analysis showed small differences.

Conclusion

BMD can be estimated with high intra- and inter-observer reliability with SECT and DECT around acetabular cups using custom software. The intra- and inter-observer agreement of DECT is superior to that of SECT and better in the cementless concept. Good intra- and inter-observer reliability can be obtained in both cemented and cementless cups using the segmentation software. SECT and DECT cannot be used interchangeably.