Inter- and intra-observer variability of qualitative visual breast-composition assessment in mammography among Japanese physicians: a first multi-institutional observer performance study in Japan

BACKGROUND

Visual assessment of mammographic breast composition remains the most common worldwide, although subjective variability limits its reproducibility. This study aimed to investigate the inter- and intra-observer variability in qualitative visual assessment of mammographic breast composition through a multi-institutional observer performance study for the first time in Japan.

METHODS

This study enrolled 10 Japanese physicians from five different institutions. They used the new Japanese breast-composition classification system 4th edition to subjectively evaluate the breast composition in 200 pairs of right and left normal mediolateral oblique mammograms (number determined using precise sample size calculations) twice, with a 1-month interval (median patient age: 59 years [range 40-69 years]). The primary endpoint of this study was the inter-observer variability using kappa (κ) value.

RESULTS

Inter-observer variability for the four and two classes of breast-composition assessment revealed moderate agreement (Fleiss' κ: first and second reading = 0.553 and 0.587, respectively) and substantial agreement (Fleiss' κ: first and second reading = 0.689 and 0.70, respectively). Intra-observer variability for the four and two classes of breast-composition assessment demonstrated substantial agreement (Cohen's κ, median = 0.758) and almost perfect agreement (Cohen's κ, median = 0.813). Assessments of consensus between the 10 physicians and the automated software Volpara® revealed slight agreement (Cohen's κ; first and second reading: 0.104 and 0.075, respectively).

CONCLUSIONS

Qualitative visual assessment of mammographic breast composition using the new Japanese classification revealed excellent intra-observer reproducibility. However, persistent inter-observer variability, presenting a challenge in establishing it as the gold standard in Japan.

Reproducibility of radiomics quality score: an intra- and inter-rater reliability study

OBJECTIVES

To investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.

METHODS

Nine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.

RESULTS

The inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).

CONCLUSIONS

Reproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.

CLINICAL RELEVANCE STATEMENT

There is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.

KEY POINTS

• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Utility of Indigenously Developed Square Grid Method for Evaluation of Tumor-Stroma Ratio and Stromal Tumor-Infiltrating Lymphocytes in Invasive Breast Carcinoma: A Pilot Study

Background & Objective

Invasive breast carcinoma (IBC) is the most commonly diagnosed cancer among women in India. The conventional visual method of evaluation of Tumor-Stroma Ratio (TSR) and Stromal Tumor-Infiltrating Lymphocytes (sTIL) appears to be subjective. The present study aims to evaluate the utility of the indigenously designed square grid method for the evaluation of tumor-stroma ratio and stromal tumor-infiltrating lymphocytes in invasive breast carcinoma by assessing the inter-observer variability.

Methods

This was a retrospective study conducted at a rural tertiary care referral institute from July 2018 to June 2020. In each case, microphotographs were taken from 10 representative fields in H&E-stained sections for evaluating TSR in low-power and sTIL in high-power. Both the parameters were evaluated employing an indigenously designed square grid applied onto microphotographs in the power-point slides by making use of principles of the Pythagorean theorem. Both parameters were separately evaluated by two pathologists. Cohen kappa statistics was the statistical tool used to analyze inter-observer variability.

Results

Thirty cases were analyzed. Invasive breast carcinoma of no special type (IBC-NST) was the most common histopathological type (26 cases (86.67%)). For TRS evaluation, a Kappa value of 0.78 suggested substantial agreement with an agreement of 91.67%. For sTIL evaluation, a Kappa value of 0.51 suggested moderate agreement with an agreement of 88.33%. The P-values were statistically highly significant (P<0.001).

Conclusion

Square grid method is a novel technique for evaluating TSR and sTIL in invasive breast carcinoma. It can be considered an example of the application of Pythagoras' theorem in Pathology.

Size measurement of lung nodules on CT: which diameter is most stable to inter-observer variability?

Indeterminate lung nodules detected on CT are common findings in the clinical practice, and the correct assessment of their size is critical for patient evaluation and management. We compared the stability of three definitions of nodule diameter (Feret's mean diameter, Martin's mean diameter and area-equivalent diameter) to inter-observer variability on a population of 336 solid nodules from 207 subjects. We found that inter-observer agreement was highest with Martin's mean diameter (intra-class correlation coefficient = 0.977, 95% Confidence interval = 0.977-0.978), followed by area-equivalent diameter (0.972, 0.971-0.973) and Feret's mean diameter (0.965, 0.964-0.966). The differences were statistically significant. In conclusion, although all the three diameter definitions achieved very good inter-observer agreement (ICC > 0.96), Martin's mean diameter was significantly better than the others. Future guidelines may consider adopting Martin's mean diameter as an alternative to the currently used Feret's (caliper) diameter for assessing the size of lung nodules on CT.

Interobserver variation in clinical target volume (CTV) delineation for stereotactic radiotherapy to non-spinal bone metastases in prostate cancer: CT, MRI and PET/CT fusion

BACKGROUND AND PURPOSE

The use of SBRT for the treatment of oligometastatic prostate cancer is increasing rapidly. While consensus guidelines are available for non-spinal bone metastases practice continues to vary widely. The aim of this study is to look at inter-observer variability in the contouring of prostate cancer non-spinal bone metastases with different imaging modalities.

MATERIALS AND METHODS

15 metastases from 13 patients treated at our centre were selected. 4 observers independently contoured clinical target volumes (CTV) on planning CT alone, planning CT with MRI fusion, planning CT with PET-CT fusion and planning CT with both MRI and PET-CT fusion combined. The mean inter-observer agreement on each modality was compared by measuring the delineated volume, generalized conformity index (CIgen), and the distance of the centre of mass (dCOM), calculated per metastasis and imaging modality.

RESULTS

Mean CTV volume delineated on planning CT with MRI and PET-CT fusion combined was significantly larger compared to other imaging modalities (p = 0.0001). CIgen showed marked variation between modalities with the highest agreement between planning CT + PET-CT (mean CIgen 0.55, range 0.32-0.73) and planning CT + MRI + PET-CT (mean CIgen 0.59, range 0.34-0.73). dCOM showed small variations between imaging modalities but a significantly shorter distance found on planning CT + PET-CT when compared with planning CT + PET-CT + MRI combined (p = 0.03).

CONCLUSIONS

Highest consistency in CTV delineation between observers was seen with planning CT + PET-CT and planning CT + PET-CT + MRI combined.

Classification regularized dimensionality reduction improves ultrasound thyroid nodule diagnostic accuracy and inter-observer consistency

PROBLEM

Convolutional Neural Networks (CNNs) for medical image analysis usually only output a probability value, providing no further information about the original image or inter-relationships between different images. Dimensionality Reduction Techniques (DRTs) are used for visualization of high dimensional medical image data, but they are not intended for discriminative classification analysis.

AIM

We develop an interactive phenotype distribution field visualization system for medical images to accurately reflect the pathological characteristics of lesions and their similarity to assist radiologists in diagnosis and medical research.

METHODS

We propose a novel method, Classification Regularized Uniform Manifold Approximation and Projection (UMAP) referred as CReUMAP, combining the advantages of CNN and DRT, to project the extracted feature vector fused with the malignant probability predicted by a CNN to a two-dimensional space, and then apply a spatial segmentation classifier trained on 2614 ultrasound images for prediction of thyroid nodule malignancy and guidance to radiologists.

RESULTS

The CReUMAP embedding correlates well with the TI-RADS categories of thyroid nodules. The parametric version that embeds external test dataset of 303 images in presence of the training data with known pathological diagnosis improves the benign and malignant nodule diagnostic accuracy (p-value = 0.016) and confidence (p-value = 1.902 × 10^-6) of eight radiologists of different experience levels significantly as well as their inter-observer agreements (kappa≥0.75). CReUMAP achieve 90.8% accuracy, 92.1% sensitivity and 88.6% specificity in test set.

CONCLUSION

CReUMAP embedding is well correlated with the pathological diagnosis of thyroid nodules, and helps radiologists achieve more accurate, confident and consistent diagnosis. It allows a medical center to generate its locally adapted embedding using an already-trained classification model in an updateable manner on an ever-growing local database as long as the extracted feature vectors and predicted diagnostic probabilities of the correspondent classification model can be outputted.

Impact of SSTR PET on Inter-Observer Variability of Target Delineation of Meningioma and the Possibility of Using Threshold-Based Segmentations in Radiation Oncology

Aim: The aim of this study was to assess the effects of including somatostatin receptor agonist (SSTR) PET imaging in meningioma radiotherapy planning by means of changes in inter-observer variability (IOV). Further, the possibility of using threshold-based delineation approaches for semiautomatic tumor volume definition was assessed. Patients and Methods: Sixteen patients with meningioma undergoing fractionated radiotherapy were delineated by five radiation oncologists. IOV was calculated by comparing each delineation to a consensus delineation, based on the simultaneous truth and performance level estimation (STAPLE) algorithm. The consensus delineation was used to adapt a threshold-based delineation, based on a maximization of the mean Dice coefficient. To test the threshold-based approach, seven patients with SSTR-positive meningioma were additionally evaluated as a validation group. Results: The average Dice coefficients for delineations based on MRI alone was 0.84 ± 0.12. For delineation based on MRI + PET, a significantly higher dice coefficient of 0.87 ± 0.08 was found (p < 0.001). The Hausdorff distance decreased from 10.96 ± 11.98 mm to 8.83 ± 12.21 mm (p < 0.001) when adding PET for the lesion delineation. The best threshold value for a threshold-based delineation was found to be 14.0% of the SUVmax, with an average Dice coefficient of 0.50 ± 0.19 compared to the consensus delineation. In the validation cohort, a Dice coefficient of 0.56 ± 0.29 and a Hausdorff coefficient of 27.15 ± 21.54 mm were found for the threshold-based approach. Conclusions: SSTR-PET added to standard imaging with CT and MRI reduces the IOV in radiotherapy planning for patients with meningioma. When using a threshold-based approach for PET-based delineation of meningioma, a relatively low threshold of 14.0% of the SUVmax was found to provide the best agreement with a consensus delineation.

A machine learning model based on readers' characteristics to predict their performances in reading screening mammograms

OBJECTIVES

Proposing a machine learning model to predict readers' performances, as measured by the area under the receiver operating characteristics curve (AUC) and lesion sensitivity, using the readers' characteristics.

METHODS

Data were collected from 905 radiologists and breast physicians who completed at least one case-set of 60 mammographic images containing 40 normal and 20 biopsy-proven cancer cases. Nine different case-sets were available. Using a questionnaire, we collected radiologists' demographic details, such as reading volume and years of experience. These characteristics along with a case set difficulty measure were fed into two ensemble of regression trees to predict the readers' AUCs and lesion sensitivities. We calculated the Pearson correlation coefficient between the predicted values by the model and the actual AUC and lesion sensitivity. The usefulness of the model to categorize readers as low and high performers based on different criteria was also evaluated. The performances of the models were evaluated using leave-one-out cross-validation.

RESULTS

The Pearson correlation coefficient between the predicted AUC and actual one was 0.60 (p < 0.001). The model's performance for differentiating the reader in the first and fourth quartile based on the AUC values was 0.86 (95% CI 0.83-0.89). The model reached an AUC of 0.91 (95% CI 0.88-0.93) for distinguishing the readers in the first quartile from the fourth one based on the lesion sensitivity.

CONCLUSION

A machine learning model can be used to categorize readers as high- or low-performing. Such model could be useful for screening programs for designing a targeted quality assurance and optimizing the double reading practice.

Diagnostic performance and inter-observer variability to differentiate between T1- and T2-stage gallbladder cancers using multi-detector row CT

PURPOSE

To evaluate the diagnostic performance and inter-observer variability of differentiating T1 and T2 gallbladder (GB) cancers using multi-detector row CT (MDCT).

METHODS

This retrospective study included 151 patients with surgically confirmed T1 (n = 49)- or T2 (n = 102)-stage GB cancer who underwent contrast-enhanced MDCT from 2016 to 2020. Five radiologists (two experienced and three less experienced) evaluated the T-stage with a confidence level calculated using a six-point scale. GB cancers were morphologically classified into three types: polypoid, polypoid with wall thickening, and wall thickening. The diagnostic performance of T-staging was assessed using receiver operating characteristic (ROC) curve analysis. Sensitivity, specificity, accuracy, positive predictive value, and negative predictive value were calculated based on a binary scale (T1 = positive). Inter-observer agreement was assessed using Fleiss κ statistics.

RESULTS

The area under the receiver operating characteristic (ROC) curve of each reviewer for T-staging ranged from 0.69 to 0.80 (median 0.77). The overall accuracy of the five radiologists was 78% (95% confidence interval [CI] 71-84%). Sensitivity was higher and specificity was lower in experienced radiologists than in less experienced radiologists (P < 0.001). The overall inter-observer agreement was fair (κ = 0.36; 95% CI 0.31, 0.41). The overall accuracy for T-stage was 63% (95% CI 48-76), 78% (95% CI 63-88), and 87% (95% CI 77-93) for polypoid, polypoid with wall thickening, and wall thickening type, respectively.

CONCLUSION

The accuracy of MDCT for differentiating T1 and T2 GB cancer is limited, and there is considerable inter-observer variability.

The dosimetric impact of axillary nodes contouring variability in breast cancer radiotherapy: an AIRO multi-institutional study

AIM

To quantify the dosimetric impact of contouring variability of axillary lymph nodes (L2, L3, L4) in breast cancer (BC) locoregional radiotherapy (RT).

MATERIALS AND METHODS

18 RT centres were asked to plan a locoregional treatment on their own planning target volume (single centre, SC-PTV) which was created by applying their institutional margins to the clinical target volume of the axillary nodes of three BC patients (P1, P2, P3) previously delineated (SC-CTV). The gold standard CTVs (GS-CTVs) of P1, P2 and P3 were developed by BC experts' consensus and validated with STAPLE algorithm. For each participating centre, the GS-PTV of each patient was created by applying the same margins as those used for the SC-CTV to SC-PTV expansion and replaced the SC-PTV in the treatment plan. Datasets were imported into MIM v6.1.7 [MIM Software Inc.], where dose-volume histograms (DVHs) were extracted and differences were analysed.

RESULTS

17/18 centres used intensity-modulated RT (IMRT). The CTV to PTV margins ranged from 0 to 10 mm (median 5 mm). No correlation was observed between GS-CTV coverage by 95% isodose and GS-PTV margins width. Doses delivered to 98% (D98) and 95% (D95) of GS-CTVs were significantly lower than those delivered to the SC-CTVs. No significant difference between SC-CTV and GS-CTV was observed in maximum dose (D2), always under 110%. Mean dose ≥ 99% of the SC-CTVs and GS-CTVs was satisfied in 84% and 50%, respectively. In less than one half of plans, GS-CTV V95% was above 90%. Breaking down the GS-CTV into the three nodal levels (L2, L3 and L4), L4 had the lowest probability to be covered by the 95% isodose.

CONCLUSIONS

Overall, GS-CTV resulted worse coverage, especially for L4. IMRT was largely used and CTV-to-PTV margins did not compensate for contouring issues. The results highlighted the need for delineation training and standardization.

Combining inter-observer variability, range and setup uncertainty in a variance-based sensitivity analysis for proton therapy

Margin concepts in proton therapy aim to ensure full dose coverage of the clinical target volume (CTV) in presence of setup and range uncertainty. Due to inter-observer variability (IOV), the CTV itself is uncertain. We present a framework to evaluate the combined impact of IOV, setup and range uncertainty in a variance-based sensitivity analysis (SA). For ten patients with skull base meningioma, the mean calculation time to perform the SA including 1.6 × 10⁴ dose recalculations was 59 min. For two patients in this dataset, IOV had a relevant impact on the estimated CTV D_95% uncertainty.

Diagnostic accuracy and inter-observer reliability of the O-RADS scoring system among staff radiologists in a North American academic clinical setting

PURPOSE

The objective of this study is to evaluate the diagnostic accuracy, interobserver variability, and common lexicon pitfalls of the ACR O-RADS scoring system among staff radiologists without prior experience to O-RADS.

MATERIALS AND METHODS

After independent review of the ACR O-RADS publications and 30 training cases, three fellowship-trained, board-certified staff radiologists scored 50 pelvic ultrasound exams using the O-RADS system. The diagnostic accuracy and area under receiver operating characteristic were analyzed for each reader. Overall agreement and pair-wise agreement between readers were also analyzed.

RESULTS

Excellent specificities (92 to 100%), NPVs (92 to 100%), and variable sensitivities (72 to 100%), PPVs (66 to 100%) were observed. Considering O-RADS 4 and O-RADS 5 as predictors of malignancy, individual reader AUC values range from 0.94 to 0.98 (p < 0.001). Overall inter-reader agreement for all 3 readers was "very good," k = 0.82 (0.73 to 0.90, 95% CI, p < 0.001). Pair-wise agreement between readers were also "very good," k = 0.86-0.92. 14 out of 150 lesions were misclassified, with the most common error being down-scoring of a solid lesion with irregular outer contours.

CONCLUSION

Even without specific training, experienced ultrasound readers can achieve excellent diagnostic performance and high inter-reader reliability with self-directed review of guidelines and cases. The study highlights the effectiveness of ACR O-RADS as a stratification tool for radiologists and supports its continued use in practice.

Inter-rater reliability between two examiners with different professional roles in the evaluation of fat infiltration in the lumbar paraspinal muscles using magnetic resonance imaging

[Purpose] To clarify the inter-rater reliability of the evaluation criteria for paraspinal muscle fat infiltration on magnetic resonance images between two examiners with different professional roles in interdisciplinary physical therapy teams. [Participants and Methods] In this retrospective study, we reviewed the clinical data of 225 patients with degenerative lumbar diseases who underwent posterior lumbar surgery at our hospital. A physical therapist and a spinal surgeon visually quantified fat infiltration of the multifidus muscles at the level of L4/5 on the preoperative magnetic resonance images of the patients using Kjaer's criteria (Grade 0: 0-10%, Grade 1: 10-50%, and Grade 2: >50%). We used the kappa coefficient to assess inter-rater reliability. [Results] The participants included 142 males and 83 females (mean age, 64.7 years; range, 21-89 years). The number of patients with grades 0/1/2 were 50/160/15, respectively, for examiner 1; and 59/155/11, respectively, for examiner 2. The kappa coefficient was 0.69, indicating a substantial agreement. [Conclusion] Our study, which is the first to assess the inter-rater reliability of Kjaer's criteria between examiners with different medical occupations, revealed that these criteria could be a reliable tool for evaluating fat infiltration in the multifidus muscles and sharing information between interdisciplinary physical therapy teams.

A whole-slide image grading benchmark and tissue classification for cervical cancer precursor lesions with inter-observer variability

The cervical cancer developing from the precancerous lesions caused by the human papillomavirus (HPV) has been one of the preventable cancers with the help of periodic screening. Cervical intraepithelial neoplasia (CIN) and squamous intraepithelial lesion (SIL) are two types of grading conventions widely accepted by pathologists. On the other hand, inter-observer variability is an important issue for final diagnosis. In this paper, a whole-slide image grading benchmark for cervical cancer precursor lesions is created and the "Uterine Cervical Cancer Database" introduced in this article is the first publicly available cervical tissue microscopy image dataset. In addition, a morphological feature representing the angle between the basal membrane (BM) and the major axis of each nucleus in the tissue is proposed. The presence of papillae of the cervical epithelium and overlapping cell problems are also discussed. Besides that, the inter-observer variability is also evaluated by thorough comparisons among decisions of pathologists, as well as the final diagnosis.

Speckle Tracking-Derived Longitudinal Strain: Validation and Influence of Scanner Settings

Speckle tracking-based strain analysis is an evolving technique for the assessment of left ventricular function. We evaluated the influence of machine settings on global longitudinal peak systolic strain (GLPSS) values in an everyday patient population (n = 35). In each patient, the four-chamber view was recorded multiple times with different machine parameters. Ejection fraction ranged between 10% and 76% and correlated well with GLPSS (r = -0.778). GLPSS was not altered systematically by modifications of gain and frame rate. Conversely, higher transducer frequencies (mean effect: 1.102%/MHz, p < 0.001, and 0.662%/MHz, p = 0.033, for harmonic and fundamental imaging frequencies, respectively) and lower sector depth (mean effect: -0.156%/cm, p < 0.001) were associated with a slight-but statistically significant-reduction in absolute GLPSS values. Intra- and inter-observer variability exhibited satisfactory repeatability. GLPSS analysis proved to be reproducible and robust in our patient cohort if common settings for adult echocardiography were applied.

Intra- and inter-observer variability in intracranial aneurysm segmentation: comparison between CT angiography (semi-automated segmentation software stroke VCAR) and digital subtraction angiography (3D rotational angiography)

PURPOSE

To compare size and morphologic features of three-dimensional aneurysm models, obtained with a semi-automated segmentation software (Stroke VCAR, GE, USA) from cerebral CT angiography (CTA) data, to three-dimensional aneurysm models obtained with digital subtraction angiography (DSA, with 3D rotational angiography acquisition-3DRA), considered as the reference standard.

METHODS

In this retrospective study, we reviewed 132 patients, with a total number of 137 intracranial aneurysm, who underwent CTA and subsequent DSA examination, supplemented with 3DRA. We compared neck length, short axis and long axis measured on 3DRA model to the same variables measured on 3D-CTA model by two blinded readers and to the automatic software dimensions. Therefore, statistics analysis assessed intra-observer and inter-observer variability and differences between patients with or without subarachnoid hemorrhage (SAH).

RESULTS

There were no significant differences in short-axis and long-axis measurements between 3D angiographic and 3D-CTA models, while comparison of neck lengths revealed a statistically significant difference, which tended to be greater for smaller neck lengths (partial volume effect and "kissing vessels" artifact). There were significant differences between manual and automatic data measured for the same three variables, and the presence of SAH did not affect aneurysm 3D reconstruction. Inter-observer agreement resulted moderate for neck length and substantial for short axis and long axis.

CONCLUSION

The examined 3D-CTA segmentation system is a reproducible procedure for aneurysm morphologic characterization and, in particular, for assessment of aneurysm sac dimensions, but considerable carefulness is required in neck length interpretation.

Inter-observer variability of cribriform architecture and percent Gleason pattern 4 in prostate cancer: relation to clinical outcome

The Grade group is an important parameter for clinical decision-making in prostate cancer. Recently, percent Gleason pattern 4 and presence of invasive cribriform and/or intraductal carcinoma (CR/IDC) have been recognized for their independent predictive value for prostate cancer outcome. There is sparse data on the inter-observer agreement for these pathologic features in practice. Our objectives were to investigate inter-observer variability of percent Gleason pattern and CR/IDC and to relate individual tumour scores to clinical outcome. Our cohort included 80 consecutive radical prostatectomies with a median follow-up 87.1 months (interquartile range 43.3-119.2), of which the slide with largest tumour volume was scored by six pathologists for Grade group (four tiers: 1, 2, 3 and 4/5), percent Gleason pattern 4 (four tiers: 0-25%, 26-50%, 51-75% and 76-100%) and presence of CR/IDC (two tiers: absent, present). The individual assignments were related to post-operative biochemical recurrence (20/80). Inter-observer agreement was substantial (Krippendorff's α 0.626) for assessment of Grade group and moderate for CR/IDC (α 0.507) and percent Gleason pattern 4 (α 0.551). For each individual pathologist, biochemical recurrence rates incremented by Grade group and presence of CR/IDC, although such relation was less clear for percent Gleason pattern 4. In conclusion, inter-observer agreement for CR/IDC and percent Gleason pattern 4 is lower than for Grade groups, indicating awareness of these features needs further improvement. Grade group and CR/IDC, but not percent Gleason pattern 4 was related to biochemical recurrence for each pathologist, indicating overall validity of individual grade assignments despite inter-observer variability.

Bowel wall thickening on computed tomography scan: Inter-observer agreement and correlation with endoscopic findings

BACKGROUND AND STUDY AIMS

Bowel wall thickening (BWT) on computed tomography (CT) has been frequently reported by radiologists. There are no clear guidelines regarding the level of thickening that is correlated with definite pathology. Radiologists usually rely on their overall subjective impression, and studies on inter-observer agreement (IOA) are lacking. This study evaluated IOA concerning BWT found on abdominal CT and identified the corresponding findings on endoscopy.

PATIENTS AND METHODS

Reports of abdominal CTs performed between January 2000 and December 2015 containing the term 'thickening' were retrieved from the radiology department database. Corresponding patients who later underwent endoscopy were included. IOA concerning BWT was evaluated using a randomly mixed sample of 80 patients with normal findings or pathological BWT on endoscopy. A search for predictive factors for the subsequent finding of malignancy on endoscopy was performed using multivariate analysis.

RESULTS

During the study period, 6142 CT scans described thickening, equivalent to a BWT prevalence of 13.7%. Ninety-one patients (mean age, 58 years) were included in the analysis. Thickening was found most commonly in the stomach (38.5%), followed by the rectum (22%) and small intestine (14%). Twenty-seven patients (29.7%) exhibited diffuse BWT, whereas 64 patients showed localised BWT (70.3%). Biopsy was performed for 64 of 91 patients with endoscopies. Among these patients, 8.8% exhibited normal findings, whereas inflammation and malignancy were discovered in 25 and 51.6% of patients, respectively, with a positive predictive value for malignancy of 0.36. The IOA concerning CT for predicting significantly pathological BWT was moderate (mean κ = 0.6). A strong association was evident between the presence of lymph nodes on CT and the presence of neoplastic lesions.

CONCLUSION

Our study strongly recommends endoscopic follow-up of patients exhibiting BWT irrespective of the thickening location, especially if it is associated with lymphadenopathy. IOA between radiologists was moderate.

Detecting and recording cardiac murmurs in clinically healthy puppies in first opinion veterinary practice at the first health check

Background

The frequency that cardiac murmurs are identified and recorded in first opinion veterinary practices at the first health check in puppies is unknown. The aims of the study were to assess the agreement between first opinion veterinary practitioners, a veterinary student and a veterinary cardiology specialist on detecting murmurs, and to establish whether abnormal auscultation findings had been recorded in the health certificates of clinically healthy puppies. The study included prospective and retrospective investigations, where the prospectively collected auscultation findings from a veterinary cardiology specialist and a trained veterinary student were compared to auscultation findings recorded by first opinion veterinary practitioners.

Results

Cardiac auscultation was performed on 331 client-owned, clinically healthy dogs at two time points: at age 34–69 days by a first opinion veterinary practitioner and at age 45–76 days, on average 9 days later, by a veterinary cardiology specialist and a trained veterinary student. Agreement among the three was compared for the presence of a murmur. The degree of inter-observer agreement was evaluated using Cohen’s kappa. Auscultation findings, as noted in the pets’ passports, from 331 puppies and 43 different first opinion veterinary practices, were retrospectively reviewed and prospectively compared with auscultation findings from a veterinary cardiology specialist. Agreement between the veterinary cardiology specialist and the first opinion veterinary practitioners was poor (ϰ = 0.01) and significantly different (P < 0.001). First opinion veterinary practitioners had recorded a cardiac murmur in only 1 of the 97 puppies in which the veterinary cardiology specialist detected a murmur. Two-hundred-and-fifty-two puppies were auscultated by both the veterinary cardiology specialist and the student. Their agreement was fair (ϰ = 0.40) and significantly different (P = 0.024). The agreement between the student and a first opinion veterinary practitioner on these 252 puppies was poor (ϰ = 0.03) and significantly different (P < 0.001).

Conclusions

This study shows that soft cardiac murmurs are rarely documented during the first veterinary health check in puppies by first opinion veterinary practitioners. Although soft murmurs may not be clinically relevant, finding and recording them is evidence of a carefully performed auscultation. Missing a non-pathological murmur is not of clinical importance; however, missing a pathological murmur could prove detrimental for the individual puppy.

Inter-observer variability of radiologists for Cambridge classification of chronic pancreatitis using CT and MRCP: results from a large multi-center study

PURPOSE

Determine inter-observer variability among radiologists in assigning Cambridge Classification (CC) of chronic pancreatitis (CP) based on magnetic resonance imaging (MRI)/magnetic resonance cholangiopancreatography (MRCP) and contrast-enhanced CT (CECT).

METHODS

Among 422 eligible subjects enrolled into the PROCEED study between 6/2017 and 8/2018, 39 were selected randomly for this study (chronic abdominal pain (n = 8; CC of 0), suspected CP (n = 22; CC of 0, 1 or 2) or definite CP (n = 9; CC of 3 or 4). Each imaging was scored by the local radiologist (LRs) and three of five central radiologists (CRs) at other consortium sites. The CRs were blinded to clinical data and site information of the participants. We compared the CC score assigned by the LR with the consensus CC score assigned by the CRs. The weighted kappa statistic (K) was used to estimate the inter-observer agreement.

RESULTS

For the majority of subjects (34/39), the group assignment by LR agreed with the consensus composite CT/MRCP score by the CRs (concordance ranging from 75 to 89% depending on cohort group). There was moderate agreement (63% and 67% agreed, respectively) between CRs and LRs in both the CT score (weighted Kappa [95% CI] = 0.56 [0.34, 0.78]; p-value = 0.57) and the MR score (weighted Kappa [95% CI] = 0.68 [0.49, 0.86]; p-value = 0.72). The composite CT/MR score showed moderate agreement (weighted Kappa [95% CI] = 0.62 [0.43, 0.81]; p-value = 0.80).

CONCLUSION

There is a high degree of concordance among radiologists for assignment of CC using MRI and CT.

Inter-observer reproducibility of classical lobular neoplasia (B3 lesions) in preoperative breast biopsies: a study of the Swiss Working Group of breast and gynecopathologists

PURPOSE

Classical type of lobular neoplasia (LN) spans a spectrum of disease, including atypical lobular hyperplasia (ALH) and lobular carcinoma in situ (LCIS), classical lobular neoplasia (LN), and the three-tiered classification of lobular intraepithelial neoplasia (LIN-1, -2, -3). This study addressed inter-observer variability of classical lobular neoplasias (LN) (B3 lesions) in preoperative breast biopsies among breast and gynecopathologists METHODS: A retrospective, observational, cross-sectional study was conducted. 40 preoperative digital images of breast core/vacuum biopsies were analyzed by eight experienced breast- and gynecopathologists. Evaluation criteria were ALH, LCIS, LN classic, LIN-1, LIN-2, LIN-3, focal B3 (one focus), extensive B3 (> one focus). Kappa-index and Chi-square tests were used for statistics. Digital scanned slides were provided to each participant. Agreement between the categories was defined as at least six of eight (cut-off 75%) concordant diagnoses.

RESULTS

The highest agreement between eight pathologists was reached using the category lobular neoplasia (LN, classical), 26/40 (65%) cases were diagnosed as such. Agreements in other categories was low or poor: 12/40 (30%) (ALH), 9/40 (22%) (LCIS), 8/40 (20%) (LIN-1), 8/40 (20%) (focal B3), 3/40 (7.5%) (LIN-2), and 2/40 (5%) (extensive B3). Chi-square-test (classical LN versus the other nomenclatures) was significant (p = 0.001137).

CONCLUSION

Our data suggest that among Swiss breast pathologists, the most reproducible diagnosis for B3 lobular lesions is the category of classical LN. These data further support lack of consistent data in retrospective studies using different terminologies. Validation of reproducible nomenclature is warranted in further studies. This information is useful especially in view of retro- and prospective data analysis with different diagnostic categories.

Multicentric Analytical and Inter-observer Comparability of Four Clinically Developed Programmed Death-ligand 1 Immunohistochemistry Assays in Advanced Clear-cell Renal Cell Carcinoma

BACKGROUND

Previous studies have suggested increased clinical benefit with inhibition of programmed death-ligand 1 (PD-L1)/programmed death-1 in patients with PD-L1-positive locally advanced/metastatic renal cell carcinoma (RCC). We examined the analytical and inter-observer comparability of PD-L1-positivity across 4 clinically developed immunohistochemistry assays in clear-cell RCC (CCRCC).

MATERIALS AND METHODS

Randomly selected archived, formalin-fixed, paraffin-embedded nephrectomy specimens from 201 patients with locally advanced CCRCC were screened using VENTANA SP142. From these, 30 cases were selected based on their tumor-infiltrating immune cell (IC) PD-L1 status (PD-L1-IC-positivity of < 1%, 1%-5%, or > 5%; 10 cases each). These cases were stained for PD-L1 using VENTANA SP142 and SP263, and DAKO 22C3 and 28-8, and scored for PD-L1 expression on IC and tumor cells (TC) by trained readers at 5 sites.

RESULTS

Adjusted mean percentages of PD-L1-IC-positivity and PD-L1-TC-positivity varied from 4.0% to 4.9% and from 1.3% to 10.7%, respectively, between assays. Inter-assay differences in PD-L1-IC-positivity were small and non-significant (P = .1938 to .9963); for PD-L1-TC-positivity, significant differences were observed between VENTANA SP142 and the other assays (P ≤ .0001) and between VENTANA SP263 and DAKO 28-8 (P = .0248). Intra-class correlation values showed moderate-to-high inter-reader agreement for each assay for PD-L1-IC-positivity and for 3 assays for PD-L1-TC-positivity.

CONCLUSIONS

In this first multicenter analytical comparison study of PD-L1 assays in CCRCC, PD-L1-positivity could be assessed reproducibly using all 4 assays for IC and for 3 of the 4 assays for TC.

Interreader variability in prostate MRI reporting using Prostate Imaging Reporting and Data System version 2.1

OBJECTIVES

To evaluate the agreement among readers with different expertise in detecting suspicious lesions at prostate multiparametric MRI using Prostate Imaging Reporting and Data System (PI-RADS) version 2.1.

METHODS

We evaluated 200 consecutive biopsy-naïve or previously negative biopsy men who underwent MRI for clinically suspected prostate cancer (PCa) between May and September 2017. Of them, 132 patients underwent prostate biopsy. Seven radiologists (four dedicated uro-radiologists and three non-dedicated abdominal radiologists) reviewed and scored all MRI examinations according to PI-RADS v2.1. Agreement on index lesion detection was evaluated with Conger's k coefficient, agreement coefficient 1 (AC1), percentage of agreement (PA), and indexes of specific positive and negative agreement. Clinical and radiological features that may influence variability were evaluated.

RESULTS

Agreement in index lesion detection among all readers was substantial (AC1 0.738; 95% CI 0.695-0.782); dedicated radiologists showed higher agreement compared with non-dedicated readers. Clinical and radiological parameters that positively influenced agreement were PSA density ≥ 0.15 ng/mL/cc, pre-MRI high risk for PCa, positivity threshold of PI-RADS score 4 + 5, PZ lesions, homogeneous signal intensity of the PZ, and subjectively easy interpretation of MRI. Positive specific agreement was significantly higher among dedicated readers, up to 93.4% (95% CI 90.7-95.4) in patients harboring csPCa. Agreement on absence of lesions was excellent for both dedicated and non-dedicated readers (respectively 85.1% [95% CI 78.4-92.3] and 82.0% [95% CI 77.2-90.1]).

CONCLUSIONS

Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1. Concordance on absence of lesions is excellent across readers' experience.

KEY POINTS

• Agreement on index lesion detection among radiologists of various experiences is substantial to excellent using PI-RADS v2.1. • Concordance between experienced readers is higher than between less-experienced readers. • Concordance on absence of lesions is excellent across readers' experience.

Inter-observer variability amongst surgeons and radiologists in assessment of Guy's Stone Score and S.T.O.N.E. nephrolithometry score: A prospective evaluation

Objective

(a) To assess the inter-observer variability amongst surgeons performing percutaneous nephrolithotomy (PCNL) and radiologists for the Guy’s Stone Score (GSS) and S.T.O.N.E. (stone size [S], tract length [T], obstruction [O], number of involved calyces [N], and essence or stone density [E]) nephrolithometry score; (b) To determine which scoring system of the two is better for predicting the stone-free rate (SFR) after PCNL.

Patients, subjects and methods

Patients undergoing PCNL between February 2016 and September 2016 were prospectively enrolled. Preoperative computed tomography was done in all patients. The GSS and S.T.O.N.E. nephrolithometry score were independently calculated by eight surgeons and four radiologists. The patients were operated on by one of the surgeons (all were consultants). The Fleiss’ κ coefficient was used to assess agreement independently between the surgeons and radiologists. Receiver operating characteristic (ROC) curves were constructed for predicting the SFR using the average of the scores of the surgeons and radiologists separately.

Results

A total of 157 patients underwent PCNL. The SFR was 71.3% (112/157 patients). The Fleiss’ κ scores ranged from 0.51 to 0.88 (overall 0.79) for the S.T.O.N.E. score and 0.53–0.91 for the GSS, suggesting moderate to very good agreement. The ROC curve for the S.T.O.N.E. nephrolithometry scores of surgeons (area under the curve [AUC] = 0.806) as well as the radiologists (AUC = 0.810) had a higher predictive value for the SFR than the GSS of the surgeons (AUC = 0.738) and the radiologists (AUC = 0.747).

Conclusion

There is overall good agreement between surgeons and radiologists for both the GSS and S.T.O.N.E. nephrolithometry score. The S.T.O.N.E. score had a higher predictive value for the SFR than the GSS.

Abbreviations

AUC: area under the curve; GSS: Guy’s Stone Score; KUB: kidneys, ureters and bladder; NCCT: non-contrast CT; PCNL: Percutaneous nephrolithotomy; ROC: receiver operating characteristic; SFR: stone-free rate; S.T.O.N.E.: stone size [S], tract length [T], obstruction [O], number of involved calyces [N], and essence or stone density [E]

Interobserver size measurement variability in part-solid lung adenocarcinoma using pre-operative computed tomography

Background

In the current lung cancer tumor-node-metastasis classification, solid tumor size is used for tumor diameter measurement as the dense component. However, measuring solid tumor size is sometimes difficult and inter-observer variability may increase, particularly in part-solid nodules with ground-glass opacity (GGO). This study aimed to investigate inter-observer size measurement variability in lung adenocarcinoma.

Methods

Of 47 patients with part-solid lung adenocarcinoma who had undergone surgery at our department from January to December 2016, five surgeons and one radiologist undertook unidimensional solid and total size tumor measurements using pre-operative axial computed tomography images, and we assessed inter-observer size measurement variability. Variability was then subclassified into five groups, according to computer tomography-identified tumor morphological characteristics, namely: (I) minimally invasive; (II) peribronchovascular; (III) spiculation/atelectasis; (IV) adjacent to cystic lesion, and; (V) diffuse consolidation and GGO.

Results

The mean inter-observer variability was 9.7 mm (solid size) and 7.7 mm (total size). Analysis of the maximum and minimum measurement size values for each patient undertaken showed that the most experienced surgeon and the radiologist measured the minimum size more frequently. To correct for differences in mean tumor diameter in each group, a comparison was made using a coefficient of variation (CV) calculated as the ratio of the standard deviation to the mean. Group I characteristics showed the largest coefficient value for variation in solid size measurement.

Conclusions

Inter-observer measurement variability for solid size was larger than for total size in lung adenocarcinoma. Large variability in group I indicated the difficulty of size measurement for low-grade malignant potential nodules such as adenocarcinoma in situ, minimally invasive adenocarcinoma, and early-stage invasive adenocarcinoma. The possibility of unavoidable size measurement variability should be recognized when deciding on surgical procedures for these diseases.

Impact of inter- and intra-observer variabilities of catheter reconstruction on multi-catheter interstitial brachytherapy of breast cancer patients

PURPOSE

The aim of this study was to evaluate inter- and intra-observer variabilities of catheter reconstruction and its dosimetric impact for multi-catheter interstitial breast cancer patients.

METHODS AND MATERIALS

In order to evaluate inter-observer variabilities (IOV) three medical physicists reconstructed the catheter traces of 13 patients. These manual reconstructions were further compared to the automatic reconstruction algorithm integrated into the planning system and one on purpose imprecise manual reconstruction. For intra-observer variabilities (IAV) repeated reconstructions of two physicists were compared for 13 patients. In total 426 catheters were considered. Keeping dwell times, dwell positions, the optimization and the normalization relative points constant the geometrical deviations between the corresponding dwell positions of the reference data set and the investigated reconstructions were evaluated. Also, the effect on the quality indices, such as coverage index (CI), dose non-uniformity ratio (DNR) or conformal index (COIN), and the exposure of the organs at risk were analyzed.

RESULTS

Over all patients and all different catheter reconstructions considered for IOV a mean deviation between the corresponding dwell positions of 0.60 ± 0.35 mm was detected. The first observer had a mean deviation of 0.54 ± 0.32 mm, whereas the second observer yielded a mean deviation of 0.58 ± 0.37 mm. The length of the catheter traces varied in the mean by 0.51 ± 0.45 mm. The mean relative deviation of the CI, DNR, COIN, mean heart dose and mean lung dose varied by 0.27 ± 0.31%, 0.0027 ± 0.0025, 0.0036 ± 0.0033, 0.024 ± 0.019%, 0.05 ± 0.11%, respectively. The skin dose (D_0.2ccm) changed in the maximum 8.52%. On average IAV reached a deviation between the corresponding dwell positions of 0.49 ± 0.30 mm. IOVs and IAVs proved to be significantly different (Wilcoxon's test p < 0.01).

CONCLUSIONS

The study proved that a repeated reconstruction of the catheter traces does not lead to large geometrical deviations or to a significant change in the dose exposure. But the lack of ground truth makes the estimation of the quality of the reconstruction challenging. A precise reconstruction mapping the reality is a necessity for the planned dose delivery. With all considered reconstruction techniques reliable quality indices for the target and the organs at risk could be obtained.

Inter-observer Variability in the Diagnosis of Proliferative Verrucous Leukoplakia: Clinical Implications for Oral and Maxillofacial Surgeon Understanding: A Collaborative Pilot Study

The use of diverse terminology may lead to inconsistent diagnosis and subsequent mistreatment of lesions within the proliferative verrucous leukoplakia (PVL) spectrum. The objectives of this study were: (a) to measure inter-observer variability between a variety of pathologists diagnosing PVL lesions; and (b) to evaluate the impact of diverse terminologies on understanding, interpretation, and subsequent treatment planning by oral and maxillofacial surgeons (OMFS). Six oral pathologists (OP) and six head and neck pathologists (HNP) reviewed 40 digitally scanned slides of PVL-type lesions. Inter-observer agreement on diagnoses was evaluated by Fleiss' kappa analysis. The most commonly used diagnostic terminologies were sent to ten OMFS to evaluate their resulting interpretations and potential follow-up treatment approaches. The overall means of the surgeons' responses were compared by Student t test. There was poor inter-observer agreement between pathologists on the diagnosis of PVL lesions (κ = 0.270), although there was good agreement (κ = 0.650) when diagnosing frankly malignant lesions. The lowest agreement was in diagnosing verrucous hyperplasia (VH) with/without dysplasia, atypical epithelial proliferation (AEP), and verrucous carcinoma (VC). The OMFS showed the lowest agreement on identical categories of non-malignant diagnoses, specifically VH and AEP. This study demonstrates a lack of standardized terminology and diagnostic criteria for the spectrum of PVL lesions. We recommend adopting standardized criteria and terminology, proposed and established by an expert panel white paper, to assist pathologists and clinicians in uniformly diagnosing and managing PVL spectrum lesions.

Inter-observer variability of clinical target volume delineation in definitive radiotherapy of neck lymph node metastases from unknown primary. A cooperative study of the Italian Association of Radiotherapy and Clinical Oncology (AIRO) Head and Neck Group

BACKGROUND

This study, promoted by Italian Association of Radiotherapy and Clinical Oncology (AIRO) Head and Neck Group, aimed to assess the current national practice of target volume delineation on a case of neck lymph node metastases from unknown primary evaluating inter-observer variability, in a setting of primary radiotherapy.

MATERIALS AND METHODS

A case of metastatic neck lymph node from occult primary was proposed to 17 radiation oncologists. A national reference RT center was identified and considered as benchmark. Participants were requested to delineate target volumes. A structured questionnaire was administered. A comparison between following parameters of the CTVs was performed: centroids distances, Dice similarity index (DSI), Jaccard index and mean distance to agreement (MDA). Volume expressed in cubic centimeters and CTVs cranio-caudal extension were evaluated.

RESULTS

Sixteen of 17 radiation oncologists recommended three CTVs dose levels. (CTV HD, CTV ID and CTV LD); CTV ID was not delineated by one of the participants and by the reference center. The distance between the reference centroid and the mean centroid of CTVs HD was 1.09 cm (0.36-3.99 cm); for CTV LD, a mean centroids distance of 2.45 (0.27-4.83 cm) was found, and for CTV HD, mean DSI is 0.48 and mean Jaccard index is 0.32 and MDA was 8.89 mm. CTV LD showed a mean DSI of 0.46, mean Jaccard index of 0.31 and MDA of 14.87 when compared to the reference.

CONCLUSION

Many aspects concerning treatment optimization of cervical nodes metastases from occult primary remain unclear, and we found a notable heterogeneity of global radiotherapy management reporting discordances both in target volume delineation and volume prescription.

RELIABILITY OF THE MICHIGAN STATE UNIVERSITY (MSU) CLASSIFICATION OF LUMBAR DISC HERNIATION

Objective:

The Michigan State University (MSU) classification of lumbar disc herniation (LDH) is periodically used by various authors to classify disc herniation. We assessed the reliability of this classification system among orthopedic residents at our institute.

Methods:

Fifty T2 axial-cut magnetic resonance images (MRI) corresponding to the level of maximal disc herniation from patients diagnosed with a single LDH were selected and distributed to six orthopedic residents. All six residents gave a specific rating for each image based on the MSU classification; in addition, three residents gave ratings on two different occasions. The degree of agreement among residents was analyzed by calculating inter-observer and intra-observer reliability using the Kappa statistic.

Results:

The inter-observer reliability among the six residents calculated as the Fleiss’ Kappa was 0.422, which indicates moderate reliability. The intra-observer reliability of three selected residents calculated by Cohen's Kappa was 0.750, 0.772, and 0.859, which indicates substantial to almost perfect reliability. Variations in ratings were frequent in images portraying a broad-based disc herniation with spinal canal stenosis.

Conclusion:

Our findings demonstrate moderate homogeneity of ratings given by residents; however, test-retest results proved the ratings to be consistent. Level of Evidence II, Diagnostic studies - investigating a diagnostic examination.

Ct radiomic features of pancreatic neuroendocrine neoplasms (panNEN) are robust against delineation uncertainty

PURPOSE

The aim of this study was to quantify the impact of CT delineation uncertainty of pancreatic neuroendocrine neoplasms (panNEN) on Radiomic features (RF).

METHODS

Thirty-one previously operated patients were considered. Three expert radiologists contoured panNEN lesions on pre-surgical high-resolution contrast-enhanced CT images and contours were transferred onto pre-contrast CT. Volume agreement was quantified by the DICE index. After images resampling and re-binning, 69 RF were extracted and the impact of inter-observer variability was assessed by Intra-Class Correlation (ICC): ICC > 0.80 was considered as a threshold for "very high" inter-observer agreement.

RESULTS

The median volume was 1.3 cc (range: 0.2-110 cc); a satisfactory inter-observer volume agreement was found (mean DICE = 0.78). Only 4 RF showed ICC < 0.80 (0.48-0.73), including asphericity and three RFs (of five) of the neighborhood intensity difference matrix (NID).

CONCLUSIONS

The impact of inter-observer variability in delineating panNEN on RF was minimum, with the exception of the NID family and asphericity, showing a moderate agreement. These results support the feasibility of studies aiming to assess CT radiomic biomarkers for panNEN.

Discrepancies of assessments in a RECIST 1.1 phase II clinical trial - association between adjudication rate and variability in images and tumors selection

Background

In imaging-based clinical trials, it is common practice to perform double reads for each image, discrepant interpretations can result from these two different evaluations. In this study we analyzed discrepancies that occurred between local investigators (LI) and blinded independent central review (BICR) by comparing reader-selected imaging scans and lesions. Our goal was to identify the causes of discrepant declarations of progressive disease (PD) between LI and BICR in a clinical trial.

Methods

We retrospectively analyzed imaging data from a RECIST 1.1-based, multi-sites, phase II clinical trial of 179 patients with adult small cell lung cancer, treated with Cabazitaxel compared to Topotecan. Any discrepancies in the determination of PD between LI and BICR readers were reviewed by a third-party adjudicator. For each imaging time point and reader, we recorded the selected target lesions, non-target lesions, and new lesions. Odds ratios were calculated to measure the association between discrepant declarations of PD and the differences in reviewed imaging scans (e.g. same imaging modality but with different reconstruction parameters) and selected lesions. Reasons for discrepancies were analyzed.

Results

The average number of target lesions found by LI and BICR was respectively 2.9 and 3.4 per patient (p < 0.05), 18.4% of these target lesions were actually non-measurable. LI and BICR performed their evaluations based on different baseline imaging scans for 59% of the patients, they selected at least one different target lesion in 85% of patients. A total of 36.7% of patients required adjudication. Reasons of adjudication included differences in 1) reporting new lesions (53.7%), 2) the measured change of the tumor burden (18.5%), and 3) the progression of non-target lesions (11.2%). The rate of discrepancy was not associated with the selection of non-measurable target lesions or with the readers’ assessment of different images. Paradoxically, more discrepancies occurred when LI and BICR selected exactly the same target lesions at baseline compared to when readers selected not exactly the same lesions.

Conclusions

For a large proportion of evaluations, LI and BICR did not select the same imaging scans and target lesions but with a limited impact on the rate of discrepancy. The majority of discrepancies were explained by the difference in detecting new lesions.

Trial Registration

ARD12166 (https://clinicaltrials.gov/ct2/show/NCT01500720).

Reproducibility and Prognostic Performance of the 1973 and 2004 World Health Organization Classifications for Grade in Non-muscle-invasive Bladder Cancer: A Multicenter Study in 328 Bladder Tumors

BACKGROUND

Histologic grade is an important prognosticator in patients with non-muscle-invasive bladder cancer (NMIBC). Currently, 2 classifications for grade are widely used; the World Health Organization (WHO) 1973 and the WHO 2004. We compare inter-observer variability of both classifications and investigate which histologic criteria cause this variability. Furthermore, the prognostic value of both classifications was assessed.

PATIENTS AND METHODS

Three pathologists reviewed 328 bladder tissue samples of 232 patients with NMIBC in a blinded manner. WHO 1973 grade, WHO 2004 grade, histologic criteria of both classifications, and T-category were evaluated. Reproducibility was analyzed using the weighted Fleiss κ, association between criteria scores and grade with the χ² test, and time-to-recurrence and time-to-progression with the log-rank test and Cox regression.

RESULTS

Reproducibility of both classifications was poor. The WHO 2004 showed better reproducibility (κ = 0.35; 95% confidence interval (CI), 0.29-0.42) compared with the WHO 1973 as a 3-tiered (κ = 0.24; 95% CI, 0.19-0.28), but not as a 2-tiered (G1 + G2 vs. G3) classification (κ = 0.36; 95% CI, 0.29-0.42). Reproducibility of individual criteria was poor (κ range, -0.05 to 0.25). All criteria were associated with grade (P < .05). After a median follow-up of 60 months, 33 of 232 and 112 of 232 patients developed progression and recurrence, respectively. In 1 out of the 3 pathologists, progression was predicted by both the WHO 1973 grade and the WHO 2004 grade in multivariable analysis. Recurrence was not predicted by grade (multivariable).

CONCLUSIONS

Reproducibility of the WHO 2004 and WHO 1973 classification for grade are poor. Scoring of individual criteria is poorly reproducible, suggesting that descriptions of these criteria for grade are not specific. The prognostic value of both the WHO 1973 and the WHO 2004 differ per pathologist.

Comparison of inter- and intra-observer variability of breast density assessments using the fourth and fifth editions of Breast Imaging Reporting and Data System

Background

Breast density is a well-known independent risk factor for breast cancer and can significantly affect the sensitivity of screening mammograms.

Objective

We aimed to evaluate the intra- and inter-observer consistencies of breast density assessments using methods outlined in the fourth and fifth editions of the American College of Radiology (ACR) Breast Imaging Reporting and Data System (BI-RADS) guidelines to determine which method is more reliable.

Materials and methods

Three radiologists with subspecialties in breast imaging defined breast density in 72 mammograms four times each: twice using the fourth edition of the ACR BI-RADS guidelines and twice using the fifth edition. The intra- and inter-observer agreements were calculated and compared for each method.

Results

The weighted kappa values for the overall intra-observer agreement were 0.955 (95% confidence interval [CI]: 0.931–0.980) and 0.938 (95% CI: 0.907–0.968) when breast densities were assessed according to criteria outlined in the fourth and fifth ACR BI-RADS editions, respectively. The difference between these values was not statistically significant (p = .4). The overall Fleiss-Cohen (quadratic) weighted kappa for inter-observer agreement were 0.623 (95% CI: 0.517–0.729) and 0.702 (95% CI: 0.589–0.815) when breast densities were assessed according to criteria outlined in the fourth and fifth ACR BI-RADS editions, respectively. The difference between these values was not statistically significant (p = .32). Similarly, there were no significant differences in the evaluation of breast density (overall) when comparing breast density assignment using criteria outlined in the fourth and fifth ACR BI-RADS edition (p = .582).

Conclusion

The ACR BI-RADS guideline is an acceptable method to classify breast density, resulting in substantial inter-observer agreements using criteria outlined in both the fourth and fifth editions. The intra-observer agreement was nearly perfect for radiologists using criteria outlined in both sets of guidelines. Moreover, although the percentage of women who were classified as having dense breasts was higher when radiologists used the fifth edition of ACR BI-RADS guidelines than when they used the fourth edition, this difference was not statistically significant.

Inter- and intra-observer variability of visual fragmented QRS scoring in ischemic and non-ischemic cardiomyopathy

BACKGROUND

Fragmented QRS (fQRS) on a 12-lead ECG has been linked with adverse outcome. However, the visual scoring of ECGs is prone to inter- and intra-observer variability.

METHODS

Five observers, two experienced and three novel, assessed fQRS in 712 digital ECGs, 100 were re-evaluated to assess intra-observer variability. Fleiss and Cohen's Kappa were calculated and compared between subgroups.

RESULTS

The inter-observer variability for assessing fQRS in all leads combined was substantial with a Kappa of 0.651. Experienced observers only had a better agreement with a Kappa of 0.823. Intra-observer variability ranged from 0.736 to 0.880. In the subgroup with ventricular pacing the inter-observer variability was even significantly larger when compared to ECGs with normal QRS duration (Kappa 0.493 vs 0.664, p<0.001).

CONCLUSION

The visual assessment of QRS fragmentation is prone to inter- and intra-observer variability, mainly influenced by the experience of the observers, the underlying rhythm and QRS morphology.

Breast MRI radiomics: comparison of computer- and human-extracted imaging phenotypes

Background

In this study, we sought to investigate if computer-extracted magnetic resonance imaging (MRI) phenotypes of breast cancer could replicate human-extracted size and Breast Imaging-Reporting and Data System (BI-RADS) imaging phenotypes using MRI data from The Cancer Genome Atlas (TCGA) project of the National Cancer Institute.

Methods

Our retrospective interpretation study involved analysis of Health Insurance Portability and Accountability Act-compliant breast MRI data from The Cancer Imaging Archive, an open-source database from the TCGA project. This study was exempt from institutional review board approval at Memorial Sloan Kettering Cancer Center and the need for informed consent was waived. Ninety-one pre-operative breast MRIs with verified invasive breast cancers were analysed. Three fellowship-trained breast radiologists evaluated the index cancer in each case according to size and the BI-RADS lexicon for shape, margin, and enhancement (human-extracted image phenotypes [HEIP]). Human inter-observer agreement was analysed by the intra-class correlation coefficient (ICC) for size and Krippendorff’s α for other measurements. Quantitative MRI radiomics of computerised three-dimensional segmentations of each cancer generated computer-extracted image phenotypes (CEIP). Spearman’s rank correlation coefficients were used to compare HEIP and CEIP.

Results

Inter-observer agreement for HEIP varied, with the highest agreement seen for size (ICC 0.679) and shape (ICC 0.527). The computer-extracted maximum linear size replicated the human measurement with p < 10⁻¹². CEIP of shape, specifically sphericity and irregularity, replicated HEIP with both p values < 0.001. CEIP did not demonstrate agreement with HEIP of tumour margin or internal enhancement.

Conclusions

Quantitative radiomics of breast cancer may replicate human-extracted tumour size and BI-RADS imaging phenotypes, thus enabling precision medicine.

The inter-observer reading variability in anti-nuclear antibodies indirect (ANA) immunofluorescence test: A multicenter evaluation and a review of the literature

Recently there has been an increase demand for Computer-Aided Diagnosis (CAD) tools to support clinicians in the field of Indirect ImmunoFluorescence (IIF), as the novel digital imaging reading approach can help to overcome the reader subjectivity. Nevertheless, a large multicenter evaluation of the inter-observer reading variability in this field is still missing. This work fills this gap as we evaluated 556 consecutive samples, for a total of 1679 images, collected in three laboratories with IIF expertise using HEp-2 cell substrate (MBL) at 1:80 screening dilution according to conventional procedures. In each laboratory, the images were blindly classified by two experts into three intensity classes: positive, negative, and weak positive. Positive and weak positive ANA-IIF results were categorized by the predominant fluorescence pattern among six main classes. Data were pairwise analyzed and the inter-observer reading variability was measured by Cohen's kappa test, revealing a pairwise agreement little further away than substantial both for fluorescence intensity and for staining pattern recognition (k=0.602 and k=0.627, respectively). We also noticed that the inter-observer reading variability decreases when it is measured with respect to a gold standard classification computed on the basis of labels assigned by the three laboratories. These data show that laboratory agreement improves using digital images and comparing each single human evaluation to potential reference data, suggesting that a solid gold standard is essential to properly make use of CAD systems in routine work lab.

Evaluation of inter-observer variability according to RECIST 1.1 and its influence on response classification in CT measurement of liver metastases

OBJECTIVE

The aim of this study is the evaluation of inter-observer variability in the measurement of liver metastases according to RECIST and its influence on response classification.

PATIENTS AND METHODS

A total of 100 radiologists measured liver target lesions, on pre- and post-chemotherapy CT scans of three patients. Each observer filled out a questionnaire about his personal and work features. The evaluations of a well experienced radiologist, considered as "the gold standard", were compared to those taken by the observers. The percentage of the observers in agreement with the reviewer, in terms of the response category and in terms of inter-observer variability, was calculated for each patient.

RESULTS

The percentage of the inter-observer agreement was elevated. Most of the observers in agreement with the reviewer were senior radiologists, while those who disagreed were junior radiologist, but this result did not reach a statistical significance. More than 30% of observers disagreed with the reviewer at least in one of the three cases.

CONCLUSIONS

RECIST measurements are reproducible on a large and heterogeneous population of radiologists. Age and expertise of the radiologist remain the most critical factors: this suggests a revision by well-experienced radiologists in clinical trials.

Cytomorphology review of 100 newly diagnosed lower-risk MDS patients in the European LeukemiaNet MDS (EUMDS) registry reveals a high inter-observer concordance

The European LeukemiaNet MDS (EUMDS) registry is collecting data of myelodysplastic syndrome (MDS) patients belonging to the IPSS low or intermediate-1 category, newly diagnosed by local cytologists. The diagnosis of MDS can be challenging, and some data report inter-observer variability with regard to the assessment of the MDS subtype. In order to ensure that correct diagnoses were made by the participating centres, blood and bone marrow slides of 10% of the first 1000 patients were reviewed by an 11-person panel of cytomorphologists. All slides were rated by at least 3 panel members (median 8 panel members; range 3-9). Marrow slides from 98 out of 105 patients were of good quality and therefore could be rated properly according to the WHO 2001 classification, including assessment of dysplastic lineages. The agreement between the reviewers whether the diagnosis was MDS or non-MDS was strong with an intra-class correlation coefficient (ICC) of 0.85. Six cases were detected not to fit the entry criteria of the registry, because they were diagnosed uniformly as CMML or AML by the panel members. The agreement by WHO 2001 classification was strong as well (ICC = 0.83). The concordance of the assessment of dysplastic lineages was substantial for megakaryopoiesis and myelopoiesis and moderate for erythropoiesis. Our data show that in general, the inter-observer agreement was high and a very low percentage of misdiagnosed cases had been entered into the EUMDS registry. Further studies including histomorphology are warranted.

Results of a multicenter survey showing interindividual variability among neurosurgeons when deciding on the radicality of surgical resection in glioblastoma highlight the need for more objective guidelines

PURPOSE

We assessed agreement among neurosurgeons on surgical approaches to individual glioblastoma patients and between their approach and those recommended by the topographical staging system described by Shinoda.

METHODS

Five neurosurgeons were provided with pre-surgical MRIs of 76 patients. They selected the surgical approach [biopsy, partial resection, or gross total resection (GTR)] that they would recommend for each patient. They were blinded to each other's response and they were told that patients were younger than 50 years old and without symptoms. Three neuroradiologists classified each case according to the Shinoda staging system.

RESULTS

Biopsy was recommended in 35.5-82.9%, partial resection in 6.6-32.9%, and GTR in 3.9-31.6% of cases. Agreement among their responses was fair (global kappa = 0.28). Nineteen patients were classified as stage I, 14 as stage II, and 43 as stage III. Agreement between the neurosurgeons and the recommendations of the staging system was poor for stage I (kappa = 0.14) and stage II (kappa = 0.02) and fair for stage III patients (kappa = 0.29). An individual analysis revealed that in contrast to the Shinoda system, neurosurgeons took into account T2/FLAIR sequences and gave greater weight to the involvement of eloquent areas.

CONCLUSIONS

The surgical approach to glioblastoma is highly variable. A staging system could be used to examine the impact of extent of resection, monitor post-operative complications, and stratify patients in clinical trials. Our findings suggest that the Shinoda staging system could be improved by including T2/FLAIR sequences and a more adequate weighting of eloquent areas.

Reproducibility of measurement of myometrial invasion in endometrial carcinoma

Myometrial invasion (MI) as a percentage (%MI), categorized into <50 or ≥50 %, is an important predictor of prognosis in endometrial carcinoma. Recent studies suggest that tumor-free distance (TFD) to serosa and the absolute depth of invasion (DOI) might be stronger predictors of prognosis. Although reproducibility is important in clinical practice for patient prognostication and treatment, reproducibility of these methods for the measurement of MI is largely unknown. One or two slides from 50 patients with FIGO stage I endometrioid endometrial carcinoma were viewed by seven gynecological pathologists, who were requested to measure %MI, TFD, and DOI. We categorized %MI as <50 % (including no MI) or ≥50 %, TFD as ≤1.75 or >1.75 mm (including no MI), ≤7 or >7 mm (including no MI), and ≤10 or >10 mm (including no MI) and DOI as <4 mm (including no MI) or ≥4 mm. Light’s kappa for multi-rater agreement was calculated. The %MI, TFD, and DOI could be measured in 88, 83, and 79 % of cases, respectively. Kappa was 0.75 for %MI, 0.77, 0.73, and 0.69 respectively for TFD with cutoffs of 1.75, 7, and 10 mm, and 0.59 for DOI. Pathologists reach substantial agreement when measuring %MI and TFD and moderate agreement when measuring DOI. The %MI can be measured in more cases than TFD and DOI. This supports the use of %MI in daily clinical practice, but future studies should compare %MI and TFD more extensively, including inter-observer variability.

Impact of coronal and sagittal views on lung gross tumor volume delineation

BACKGROUND AND PURPOSE

To study the impact of coronal and sagittal views (CSV) on the gross tumor volume (GTV) delineation on CT and matched PET/CT scans in non-small cell lung cancer.

MATERIAL AND METHODS

GTV delineations were performed by 11 experienced radiation oncologists on CT and PET/CT in 22 patients. Two tumor groups were defined: Group I: Primary tumors surrounded by lung or visceral pleura, without venous invasion, and without large extensions to the chest wall or the mediastinum. Group II: Tumors invading the hilar region, heart, large vessels, pericardium, and the mediastinum and/or associated with atelectasis. Tumor volumes and inter-observers variations (SD) were calculated and compared according to the use of axial view only (AW), axial/coronal/sagittal views (ACSW) and ACSW/PET (ACSWP).

RESULTS

CSV were not frequently used (57.4% out of 242 delineations on CT). For group I, ACSW didn't improve significantly mean GTVs. SDs were small on CT and on PET (SD=0.3cm). For group II, ACSW had 27-46% smaller observer variation (mean SD=0.7cm) than AW (mean SD=1.1cm). The smaller observer variation of ACSW users was associated with, on average, a 40% smaller delineated volume (p=0.038). Mean GTV of ACSWP was 21% larger than mean GTV of ACSW on CT.

CONCLUSIONS

For smaller lung tumors surrounded by healthy lung tissue the effect of multiple axis delineation is limited. However, application of coronal and sagittal windows is highly beneficial for delineation of more complex tumors, with atelectasis and/or pathological lymph nodes even if PET is used.

Inter-observer and inter-examination variability of manual vertebral bone attenuation measurements on computed tomography

Objective

To determine inter-observer and inter-examination variability of manual attenuation measurements of the vertebrae in low-dose unenhanced chest computed tomography (CT).

Methods

Three hundred and sixty-seven lung cancer screening trial participants who underwent baseline and repeat unenhanced low-dose CT after 3 months because of an indeterminate lung nodule were included. The CT attenuation value of the first lumbar vertebrae (L1) was measured in all CTs by one observer to obtain inter-examination reliability. Six observers performed measurements in 100 randomly selected CTs to determine agreement with limits of agreement and Bland-Altman plots and reliability with intraclass correlation coefficients (ICCs). Reclassification analyses were performed using a threshold of 110 HU to define osteoporosis.

Results

Inter-examination reliability was excellent with an ICC of 0.92 (p < 0.001). Inter-examination limits of agreement ranged from -26 to 28 HU with a mean difference of 1 ± 14 HU. Inter-observer reliability ICCs ranged from 0.70 to 0.91. Inter-examination variability led to 11.2 % reclassification of participants and inter-observer variability led to 22.1 % reclassification.

Conclusions

Vertebral attenuation values can be manually quantified with good to excellent inter-examination and inter-observer reliability on unenhanced low-dose chest CT. This information is valuable for early detection of osteoporosis on low-dose chest CT.

Key Points

• Vertebral attenuation values can be manually quantified on low-dose unenhanced CT reliably.

• Vertebral attenuation measurements may be helpful in detecting subclinical low bone density.

• This could become of importance in the detection of osteoporosis.

Electronic supplementary material

The online version of this article (doi:10.1007/s00330-015-4145-x) contains supplementary material, which is available to authorized users.

Variability in Interpretation of Ultrasound Elastography and Gray-Scale Ultrasound in Assessing Thyroid Nodules

The aim of this study was to validate inter-observer variability for strain ultrasound elastography (USE) and to compare the diagnostic performance of a combination of gray-scale ultrasound (US) and USE with that of gray-scale US. Three observers from different institutions evaluated gray-scale US images and USE video files of 443 cytopathologically proven benign or malignant thyroid nodules over a 3-mo period. Inter-observer variability did not statistically differ between USE using the Asteria criteria and gray-scale US; however, USE using the Rago criteria had the lowest inter-observer agreement (p < 0.043). For all three observers, sensitivity was increased by adding USE to gray-scale US (81.3%-88.3%, 75.4%-85.4%) compared with gray-scale US (70.4%-80.8%). Specificity was decreased by adding USE to gray-scale US (51.7%-59.1%, 59.1%-73.9%) compared with gray-scale US (69.0%-82.8%). USE and gray-scale US had comparable inter-observer variability. However, on addition of USE to gray-scale US, the additional diagnostic yield was limited compared with that of gray-scale US alone.

Inter-Observer Variability in the Interpretation of Endoscopic Mucosal Resection Specimens of Esophageal Adenocarcinoma: Interpretation of ER specimens

INTRODUCTION

Endoscopic resection (ER) has revolutionized the staging and therapy of superficial esophageal adenocarcinoma. Pathologic evaluation allows an assessment of the risk of lymph node metastases based on tumor characteristics. The aim of this study was to assess the inter-observer variability in pathologic assessment of ER specimens of esophageal adenocarcinoma.

METHODS

We performed a retrospective study on ER specimens of superficial esophageal adenocarcinoma from four US institutions. Original endoscopic resection slides were re-reviewed by two blinded, experienced (study) gastrointestinal pathologists for the depth of tumor invasion, tumor grade, and the presence of lymphovascular invasion (LVI). Discordance was considered present only when both study pathologists disagreed with the original report.

RESULTS

There were 25 ER specimens reviewed for this study, and discordance occurred in 12 of the 25 specimens (48%) for the depth of tumor invasion. In most cases (83%), the discordance was related to overstaging a true T1a lesion. We found that only 38% of true T1a lesions were correctly staged for depth of invasion. Less commonly discordance was secondary to understaging a true T1b lesion. There was concordance between the two study pathologists in 22/25 cases (88%) on the depth of invasion. Discordance was present for tumor grade in 8/18 cases (44%) and for LVI in 4/16 cases (25%). Concordance between the study pathologists was 80% for tumor grade and 88% for LVI.

CONCLUSIONS

There was an alarmingly high rate of discordance (48%) between the study pathologists and the original pathology assessment for the depth of tumor invasion in ER specimens. This was particularly common for lesions called T1b on the original pathology report. Since critical decisions are made regarding esophageal preservation or esophagectomy on the basis of the pathologic interpretations of ER specimens, it behooves surgeons to understand the inter-observer variability. Review of ER specimens by an experienced GI pathologist is recommended to ensure that patients receive the appropriate treatment for superficial esophageal adenocarcinoma.

Inter- and intra-observer variability in the classification of extracapsular extension in p16 positive oropharyngeal squamous cell carcinoma nodal metastases

OBJECTIVE

Discern inter- and intra-observer variability in the classification of extracapsular extension (ECE) in p16+ oropharyngeal (OP) SCC comparing pathologists' own criteria versus those of a well-defined classification system.

METHODS

Five pathologists reviewed 50 digitally scanned nodal metastasis slides in three Rounds. Round One was by their own criteria as ECE present or absent, and Rounds Two and Three were with a defined ECE system: Grade 0 (no ECE), 0c (no ECE - thick capsule; no infiltration), 1 (ECE - cells beyond capsule), and 2 (soft tissue metastasis - cells in soft tissue without residual node). Round Three assessed intra-observer variability after an 8 month washout period.

RESULTS

In Round One, all five agreed on only 48% of cases (n=24). Fleiss's Kappa value was 0.508 (95% CI: 0.357-0.644). For Rounds Two and Three, Grades 0 and 0c and Grades 1 and 2 were separately grouped as ECE absent or present. In Round Two, all five agreed on 68% of cases (n=34). Fleiss' Kappa was 0.635 (95% CI: 0.472-0.783), indicating statistically significantly better agreement. In Round Three, all five agreed on 64% of cases (n=32) giving a Fleiss's Kappa of 0.639. Pathologists agreed with their prior reads in approximately 90% of cases (average n=45.4, range n=42-49), an average intra-observer Cohen's Kappa of 0.8 (range: 0.68-0.95). Inter- and intra-observer variability rates for classification of soft tissue metastasis (ECE2) were substantially worse.

CONCLUSION

There is substantial inter-, and modest intra-, observer variability among head and neck pathologists for ECE in p16+ OPSCC, which is modestly improved by a defined system.

Inter-observer variability of response evaluation criteria for hepatocellular carcinoma treated with chemoembolization

BACKGROUND

Data comparing EASL and mRECIST criteria for response evaluation in treatment of hepatocellular carcinoma are rare. We evaluated inter-observer variability by these two response evaluation criteria in treatment-naïve patients undergoing chemoembolization.

METHODS

For 133 patients undergoing chemoembolization, two radiologists independently measured sum of bi-dimensional and uni-dimensional diameters at baseline using both EASL criteria and mRECIST, and their changes on first follow-up for up to 5 target lesions.

RESULTS

Concordance correlation coefficients for sum of bi-dimensional and uni-dimensional diameters at baseline between two observers were 0.992 and 0.988, respectively. However, those for their changes on follow-up were 0.865 and 0.877, respectively. Similarly, mean differences in sum of bi-dimensional and uni-dimensional diameters at baseline between two observers were small; -0.455 and 0.079 cm, respectively. However, mean differences in changes (%) in sum of bi-dimensional and uni-dimensional diameters on first follow-up between observers increased by -9.715% and -9.320%, respectively. Regarding tumour numbers, kappa-value between observers was 0.942. For treatment response (complete or partial response, stable disease and progression), kappa-value was 0.941 by both criteria. When only up to two target lesions were assessed, kappa-value was 1.000 by both criteria.

CONCLUSIONS

Inter-observer agreements using both response evaluation criteria were excellent, especially when up to two targets were assessed.

New methodology for analysing and increasing the cost-efficiency of environmental monitoring networks

This study focuses on the coastal monitoring network established in the scope of WFD implementation. The objective of this network was to provide an ecological assessment of Valencian coastal waters. After three years, sufficient data had been collected to enable us to analyse and explore ways to increase the network's efficiency. A methodology was developed to select the best subset of sampling stations to be surveyed. This method was approached from the perspective of an inter-observer variability problem. In order to compare the concordance between the k-observers and the reference observer, two measures were considered: euclidean distance, and interclass correlation coefficient. The obtained results confirm that the current network can be reduced by over 50% and still guarantee accurate results. This methodology (not limited by indicators, geographically, or by type of water body) could be applied to different environmental monitoring networks and could significantly decrease the efforts and costs required by the WFD.

Impedance pH Monitoring: Intra-observer and Inter-observer Agreement and Usefulness of a Rapid Analysis of Symptom Reflux Association

BACKGROUND/AIMS

Symptom reflux association analysis is especially helpful for evaluation and management of proton pump inhibitor (PPI) re-fractory patients. An accurate calculation requires manual editing of 24-hour multichannel intraluminal impedance-pH (MII-pH) tracings after automatic analysis. Intra- and inter-observer agreement as well as reliability of rapid editing confined to the time around symptomatic episodes are unknown. Aim of this study was to explore these topics in a prospective multicenter study.

METHODS

Forty consecutive patients who were off PPI therapy underwent MII-pH recordings. After automatic analysis, their tracings were anonymized and randomized. Three experienced observers, each one trained in a different European center, independently per-formed manual editing of 24-hour tracings on 2 separate occasions. Values of symptom index and symptom association proba-bility for acid and non acid reflux were transformed into binary response (i.e., positive or negative).

RESULTS

Intra-observer agreement on symptom reflux association was 92.5% to 100.0% for acid and 85.0% to 97.5% for non-acid reflux. Inter-observer agreement was 100.0% for acid and 82.5% to 95.0% for non-acid reflux. Values for symptom index and symptom association probability were similar. Concordance between 24-hour and rapid (2 minutes-window before each symp-tomatic episode) editings for symptom reflux association occured in 39 to 40 patients (acid) and in 37 to 40 (non-acid), de-pending on the observer.

CONCLUSIONS

Intra- and inter-observer agreement in classifying patients with or without symptom reflux association at manual editing of 24-hour tracings was high, especially for acid reflux. Classifying patients according to a rapid editing showed excellent con-cordance with the 24-hour one and can be adopted in clinical practice.

CT-based patient modeling for head and neck hyperthermia treatment planning: manual versus automatic normal-tissue-segmentation

BACKGROUND AND PURPOSE

Clinical trials have shown that hyperthermia, as adjuvant to radiotherapy and/or chemotherapy, improves treatment of patients with locally advanced or recurrent head and neck (H&N) carcinoma. Hyperthermia treatment planning (HTP) guided H&N hyperthermia is being investigated, which requires patient specific 3D patient models derived from Computed Tomography (CT)-images. To decide whether a recently developed automatic-segmentation algorithm can be introduced in the clinic, we compared the impact of manual- and automatic normal-tissue-segmentation variations on HTP quality.

MATERIAL AND METHODS

CT images of seven patients were segmented automatically and manually by four observers, to study inter-observer and intra-observer geometrical variation. To determine the impact of this variation on HTP quality, HTP was performed using the automatic and manual segmentation of each observer, for each patient. This impact was compared to other sources of patient model uncertainties, i.e. varying gridsizes and dielectric tissue properties.

RESULTS

Despite geometrical variations, manual and automatic generated 3D patient models resulted in an equal, i.e. 1%, variation in HTP quality. This variation was minor with respect to the total of other sources of patient model uncertainties, i.e. 11.7%.

CONCLUSIONS

Automatically generated 3D patient models can be introduced in the clinic for H&N HTP.