New methods give better estimates of changes in diagnostic accuracy when prior information is provided

Assessing Digital Breast Tomosynthesis Impact on Early Cancer Detection: Insights from Consecutive Screening

Background Analysis of how digital breast tomosynthesis (DBT) screening affects consecutive screening performance is important to estimate its future value in screening. Purpose To evaluate whether DBT contributes to early detection of breast cancer by assessing cancer detection rates (CDRs), including the fraction of invasive cancers and cancer subtypes in consecutive routine digital mammography (DM). Materials and Methods The paired prospective Malmö Breast Tomosynthesis Screening Trial (MBTST) was performed between January 2010 and February 2015. Participating women underwent one-view DBT and two-view DM at one screening occasion. In this secondary analysis, women were followed up through their first (DM1) and second (DM2) consecutive two-view DM screening rounds after MBTST participation. Cancer diagnoses were identified by referencing records. A logistic regression model, adjusted for age, was used to calculate the odds of luminal A-like cancers with use of the MBTST as reference. Results There were 14 848 final participants in the MBTST (median age, 57 years [IQR, 49-65 years]). Of those, 12 876 women were screened in DM1 (median age, 58 years [IQR, 50-66 years]) and 10 883 were screened in DM2 (median age, 59 years [IQR, 51-67 years]). Compared with CDRs in the trial of 6.5 of 1000 women (95% CI: 5.2, 7.9) for DM and 8.7 of 1000 women (95% CI: 7.3, 10.3) for DBT, the CDR was lower in DM1 (4.6 of 1000 women [95% CI: 3.6, 5.9]) and DM2 (5.3 of 1000 women [95% CI: 4.1, 6.9]). The proportion of invasive cancers was 84.9% (118 of 139 cancers) in the MBTST; the corresponding numbers were 66% (39 of 59 cancers) for DM1 and 83% (50 of 60 cancers) for DM2. The odds of luminal A-like cancers were lower in DM1 at 0.28 (95% CI: 0.12, 0.66 [P = .004]) but not in DM2 at 0.80 (95% CI: 0.40, 1.58 [P = .52]) versus screening in the MBTST. Conclusion CDR and the fraction of invasive cancers were lower in DM1 and then increased in DM2 following the MBTST, indicating earlier cancer detection mainly due to increased detection of luminal A-like cancers in DBT screening. Clinical trials registration no. NCT01091545 © RSNA, 2024 See also the editorial by Hooley and Philpotts in this issue.

The Effect of Clinical History on Diagnostic Imaging Interpretation - A Systematic Review

RATIONALE AND OBJECTIVES

To provide updated information on the effect of clinical history on diagnostic image interpretation and to provide study methodology and design recommendations for future studies assessing the effect of clinical history on diagnostic image performance.

MATERIALS AND METHODS

A literature search of Medline, Embase, Scopus, Web of Science, and the Cochrane Central Register of Controlled Trials (CENTRAL) databases was conducted from database inception to July 21, 2020. Studies comparing diagnostic imaging performance with and without clinical history, using observers reading images under both conditions that used an independent reference standard were included.

RESULTS

Twenty-two studies met the inclusion criteria, with 15 showing clinical history improved diagnostic performance. One study reported a decrease in diagnostic performance with clinical history and the remaining six studies found no significant change in performance. Two studies used the free response paradigm with both reporting clinical history increased location sensitivity, decreased specificity and had no overall change in diagnostic performance. The disease spectrum of included cases was largely unreported and a balanced reading design was not used in 19 studies.

CONCLUSION

Most published studies found that clinical history improved diagnostic performance. More recent studies accounting for abnormality location and multiple abnormalities showed an increase in false positives and no significant change in overall diagnostic performance with clinical history.

Diagnostics in Patients Suspect for Breast Cancer in The Netherlands

The goal of this study was to describe the variation in hospital-based diagnostic care activities for patients with symptomatology suspect for breast cancer in The Netherlands. Two cohorts were included: the ‘benign’ cohort (30,334 women suspected of, but without breast cancer) and the ‘malignant’ cohort (2236 breast cancer patients). Hospital-based financial data was combined with tumor data (malignant cohort) from The Netherlands Cancer Registry. Patterns within diagnostic pathways were analyzed. Factors influencing the number of visits and number of diagnostic care activities until diagnosis were identified in the malignant cohort with multivariable Cox and Poisson regression models. Compared to patients with benign diagnosis, patients with malignant disease received their diagnosis less frequently in one day, after an equal average number of hospital visits and higher average number of diagnostic activities. Factors increasing the number of diagnostic care activities were the following: lower age and higher cM-and cN-stages. Factors increasing the number of days until (malignant) diagnosis were as follows: higher BIRADS-score, screen-detected and higher cN-and cT-stages. Hospital of diagnosis influenced both number of activities and days to diagnosis. The diagnostic care pathway of patients with malignant disease required more time and diagnostic activities than benign disease and depends on hospital, tumor and patient characteristics.

A systematic review classifies sources of bias and variation in diagnostic test accuracy studies

OBJECTIVE

To classify the sources of bias and variation and to provide an updated summary of the evidence of the effects of each source of bias and variation.

STUDY DESIGN AND SETTING

We conducted a systematic review of studies of any design with the main objective of addressing bias or variation in the results of diagnostic accuracy studies. We searched MEDLINE, EMBASE, BIOSIS, the Cochrane Methodology Register, and Database of Abstracts of Reviews of Effects (DARE) from 2001 to October 2011. Citation searches based on three key papers were conducted, and studies from our previous review (search to 2001) were eligible. One reviewer extracted data on the study design, objective, sources of bias and/or variation, and results. A second reviewer checked the extraction.

RESULTS

We summarized the number of studies providing evidence of an effect arising from each source of bias and variation on the estimates of sensitivity, specificity, and overall accuracy.

CONCLUSIONS

We found consistent evidence for the effects of case-control design, observer variability, availability of clinical information, reference standard, partial and differential verification bias, demographic features, and disease prevalence and severity. Effects were generally stronger for sensitivity than for specificity. Evidence for other sources of bias and variation was limited.

Comparison of Breast Cancer Screening Results in Korean Middle-Aged Women: A Hospital-based Prospective Cohort Study

Objectives

The aim of this hospital-based prospective study was to evaluate the diagnostic ability of breast cancer screening in Korean middle-aged women using age, ultrasonography, mammography, and magnification mammography, which are commonly used in most hospitals.

Methods

A total of 21 patents were examined using ultrasonography, mammography, and magnification mammography, and their data were prospectively analyzed from August 2011 to March 2013. All patients were divided into benign and malignant groups and the screening results were classified using the American College of Radiology Breast Imaging Reporting and Data System (BI-RADS). The final pathology report was used as the reference standard and the sensitivity and specificity of ultrasonography, mammography, and magnification mammography were evaluated using receiver-operating characteristics (ROC) analysis.

Results

The analysis included 21 patients who underwent biopsy. Among them, three (14.3%) were positive and 18 (85.7%) negative for breast cancer. The average age was 50.5 years (range = 38–61 years). The sensitivity was the same for ultrasonography and magnification mammography and the specificity of magnification mammography was higher than that of ultrasonography. The highest area under the ROC curve (AUC) was observed in the combination of age and magnification mammography (1.000) and the decreasing order of AUC in others was magnification mammography (0.833), ultrasonography (0.787), mammography (0.667), and age (0.648).

Conclusions

In Korean women, the diagnostic accuracy of magnification mammography was better than that of ultrasonography and mammography. The combination of age and magnification mammography increased the sensitivity and diagnostic accuracy.

The effect of clinical history on accuracy of electrocardiograph interpretation among doctors working in emergency departments

OBJECTIVE

To investigate whether bias in clinical history affects accuracy of electrocardiograph (ECG) interpretation among doctors working in emergency departments.

DESIGN AND SETTING

Observational study conducted at four teaching hospitals in Sydney from May to September 2011.

PARTICIPANTS

Participants interpreted 30 ECGs representing 10 diagnoses. ECGs were provided with positively biased history (suggestive of the correct diagnosis), negatively biased history (suggestive of an alternative diagnosis) or no history.

MAIN OUTCOME MEASURES

Accuracy of ECG interpretation, measured as a score out of 10 (for each category of clinical history) and as a percentage of correctly interpreted ECGs.

RESULTS

Of 307 doctors who were sent a recruitment email for the study, 132 participated (43%). The overall mean accuracy of ECG interpretation was 52% (95% CI, 50%-53%). For junior doctors, mean accuracy was 42% (95% CI, 40%-44%); for senior doctors, it was 65% (95% CI, 62%-67%). In adjusted models, the mean predicted score for senior doctors provided with no history was 6.25 (95% CI, 5.90-6.62) with junior doctors obtaining mean scores 34% lower than senior doctors (95% CI, 29%-40%; P < 0.001). Compared with no history, positively biased history was associated with 42% higher mean scores (95% CI, 35%-49%; P < 0.001) and negatively biased history was associated with 34% lower mean scores (95% CI, 29%-39%; P < 0.001).

CONCLUSION

Bias in clinical history significantly influenced the accuracy of ECG interpretation. Strategies that reduce the detrimental impact of cognitive bias and improved ECG training for doctors are recommended.

The impact of clinical data on the evaluation of tibial fracture healing

Background

Radiographic healing is a common outcome measure in orthopedic trials and adjudication by outcome assessors is often conducted on the basis of plain films alone. The degree to which this process reflects clinical practice, in which both plain films and clinical notes are available, is uncertain. We explored the effect of adding clinical notes to radiographs in the adjudication process of a feasibility trial of tibial shaft fractures.

Methods

Radiographic and clinical data from a multicenter randomized controlled trial of 51 patients with operatively treated tibial fractures formed the basis of the study data. At the completion of the trial, serial radiographs (anteroposterior and lateral) were independently evaluated for progression of fracture healing, defined as bridging of at least 3 of 4 cortices, by an adjudication committee comprised of 3 blinded orthopaedic trauma surgeons. Immediately after determination of radiographic time to healing, each surgeon was provided with clinical notes associated with each radiographic follow up visit and asked to re-visit their initial impression. Consensus was achieved for both adjudications. We calculated the percentage of time to healing consensus decisions that changed after evaluation of clinical notes. We further examined the contents of clinical notes and their relative influence on the committee's decisions.

Results

47 of 51 patients were determined to have healed radiographically during the trial follow-up period, and consideration of clinical notes resulted in a change of 40% (19 of 47) of time to healing consensus decisions; however, revised decisions were equally likely to support an earlier or a later time to healing. Clinical notes that resulted in a change to either a 'healed' or a 'not healed' decision contained significantly more comments of either pain resolution or deterioration, respectively, resumption of or failure to resume weightbearing, or either return or no return to work/pre-injury activities (p < 0.001).

Conclusions

The addition of clinical notes to the adjudication of radiographic fracture healing changed the outcome decision in a substantial number of cases. Orthopedic trialists should consider the addition of clinical notes to adjudication material in studies of fracture healing in order to enhance the generalizability of their results.

Trial Registration

The TRUST trial was registered [ID NCT00667849] at http://clinicaltrials.gov/ct2/show/NCT00667849

Systematic review: bias in imaging studies - the effect of manipulating clinical context, recall bias and reporting intensity

OBJECTIVES

To perform a systematic review of diagnostic test accuracy studies which manipulate or investigate the context of interpretation. In particular, those which modify or conceal sample characteristics (e.g. disease prevalence or reporting intensity) or research setting ("laboratory" versus "field"). We also investigated recall bias.

METHODS

We searched the biomedical literature to March 2010 using 3 complementary strategies. Inclusion criteria were: imaging studies quantifying the effect on diagnosis of modifying the context of observers' interpretations, varying disease prevalence, concealing sample characteristics, reporting intensity and recall bias.

RESULTS

11247 abstracts were reviewed, 201 full texts examined and 12 ultimately included. There were 5 to 9520 patients and 2 to 129 observers per study. Nine studies investigated clinical review bias of sample level information. Only 3 studies investigated prevalence, 2 of which investigated maximum enrichment well below the levels often used by researchers. We identified no research specifically directed at concealing disease prevalence. Available research found no evidence of recall bias or "washout" on study results.

CONCLUSIONS

Several sources of bias central to the design of diagnostic test accuracy studies are poorly researched; the implications for evidence-based-practice remain uncertain. Research is suggested to guide methodological design, particularly in the context of screening.

KEY POINTS

Imaging research studies often ignore the possible effect of disease prevalence It is unclear how the expectation of disease influences radiological interpretation The potential effect of observer recall bias is poorly researched Such factors might introduce bias into radiological research methodology This systematic review attempts to illustrate these points.

Evaluating agreement: conducting a reliability study

Instruments that are useful in clinical or research practice will, when the object of measurement is stable, yield similar results when applied at different times, in different situations, or by different users. Studies that measure the relation of differences between patients or subjects and measurement error (reliability studies) are becoming increasingly common in the orthopaedic literature. In this paper, we identify common aspects of reliability studies and suggest features that improve the reader's confidence in the results. One concept serves as the foundation for all further consideration: in order for a reliability study to be relevant, the patients, raters, and test administration in the study must be similar to the clinical or research context in which the instrument will be used. We introduce the statistical measures that readers will most commonly encounter in reliability studies, and we suggest an approach to sample-size estimation. Readers interested in critically appraising reliability studies or in developing their own reliability studies may find this review helpful.

Probabilistic computer model developed from clinical data in national mammography database format to classify mammographic findings

PURPOSE

To determine whether a Bayesian network trained on a large database of patient demographic risk factors and radiologist-observed findings from consecutive clinical mammography examinations can exceed radiologist performance in the classification of mammographic findings as benign or malignant.

MATERIALS AND METHODS

The institutional review board exempted this HIPAA-compliant retrospective study from requiring informed consent. Structured reports from 48 744 consecutive pooled screening and diagnostic mammography examinations in 18 269 patients from April 5, 1999 to February 9, 2004 were collected. Mammographic findings were matched with a state cancer registry, which served as the reference standard. By using 10-fold cross validation, the Bayesian network was tested and trained to estimate breast cancer risk by using demographic risk factors (age, family and personal history of breast cancer, and use of hormone replacement therapy) and mammographic findings recorded in the Breast Imaging Reporting and Data System lexicon. The performance of radiologists compared with the Bayesian network was evaluated by using area under the receiver operating characteristic curve (AUC), sensitivity, and specificity.

RESULTS

The Bayesian network significantly exceeded the performance of interpreting radiologists in terms of AUC (0.960 vs 0.939, P = .002), sensitivity (90.0% vs 85.3%, P < .001), and specificity (93.0% vs 88.1%, P < .001).

CONCLUSION

On the basis of prospectively collected variables, the evaluated Bayesian network can predict the probability of breast cancer and exceed interpreting radiologist performance. Bayesian networks may help radiologists improve mammographic interpretation.

Inter-observer agreement for abdominal CT in unselected patients with acute abdominal pain

The level of inter-observer agreement of abdominal computed tomography (CT) in unselected patients presenting with acute abdominal pain at the Emergency Department (ED) was evaluated. Two hundred consecutive patients with acute abdominal pain were prospectively included. Multi-slice CT was performed in all patients with intravenous contrast medium only. Three radiologists independently read all CT examinations. They recorded specific radiological features and a final diagnosis on a case record form. We calculated the proportion of agreement and kappa values, for overall, urgent and frequently occurring diagnoses. The mean age of the evaluated patients was 46 years (range 19-94), of which 54% were women. Overall agreement on diagnoses was good, with a median kappa of 0.66. Kappa values for specific urgent diagnoses were excellent, with median kappa values of 0.84, 0.90 and 0.81, for appendicitis, diverticulitis and bowel obstruction, respectively. Abdominal CT has good inter-observer agreement in unselected patients with acute abdominal pain at the ED, with excellent agreement for specific urgent diagnoses as diverticulitis and appendicitis.