Systematic review: bias in imaging studies - the effect of manipulating clinical context, recall bias and reporting intensity

Spectrum of Cognitive Biases in Diagnostic Radiology

Cognitive biases are systematic thought processes involving the use of a filter of personal experiences and preferences arising from the tendency of the human brain to simplify information processing, especially when taking in vast amounts of data such as from imaging studies. These biases encompass a wide spectrum of thought processes and frequently overlap in their concepts, with multiple biases usually in operation when interpretive and perceptual errors occur in radiology. The authors review the gamut of cognitive biases that occur in radiology. These biases are organized according to their expected stage of occurrence while the radiologist reads and interprets an imaging study. In addition, the authors propose several additional cognitive biases that have not yet, to their knowledge, been defined in the radiologic literature but are applicable to diagnostic radiology. Case examples are used to illustrate potential biases and their impact, with emergency radiology serving as the clinical paradigm, given the associated high imaging volumes, wide diversity of imaging examinations, and rapid pace, which can further increase a radiologist's reliance on biases and heuristics. Potential strategies to recognize and overcome one's personal biases at each stage of image interpretation are also discussed. Awareness of such biases and their unintended effects on imaging interpretations and patient outcomes may help make radiologists cognizant of their own biases that can result in diagnostic errors. Identification of cognitive bias in departmental and systematic quality improvement practices may represent another tool to prevent diagnostic errors in radiology. ©RSNA, 2024 See the invited commentary by Larson in this issue.

Doug Altman, medical statistician par excellence: What can radiologists learn from his legacy?

This narrative review describes our experience of working with Doug Altman, the most highly cited medical statistician in the world. Doug was particularly interested in diagnostics, and imaging studies in particular. We describe how his insights helped improve our own radiological research studies and we provide advice for other researchers hoping to improve their own research practice.

Utility of gadolinium for identifying the malignant potential of pancreatic cystic lesions

PURPOSE

To determine if gadolinium is necessary for the diagnosis of a pancreatic cystic lesion (PCL) as benign or malignant by assessing inter- and intra-observer agreement and diagnostic accuracy for the presence of worrisome features/high-risk stigmata on non-contrast MRI compared to MRI with and without contrast, with cytopathology as a reference standard.

METHODS

The institutional database was searched to identify consecutive patients that underwent EUS/FNA or surgical resection of an asymptomatic PCL performed from 01/01/2015 to 01/01/2019. Two abdominal radiologists independently evaluated PCLs on MRI with all sequences except for contrast-enhanced sequences followed by a second reading with data from the entire MRI including pre- and post-contrast sequences. Cyst size, growth, and the presence of worrisome features/high-risk stigmata were assessed for each cyst on both datasets.

RESULTS

There were 87 patients with 87 pancreatic cysts; 76(87.4%) were benign and 11 (12.7%) were malignant. The presence of any worrisome features/high-risk stigmata for reader 1 was concordant on both MRIs in 95.4% (83/87; k = 0.874) of cases and for reader 2 was concordant in 96.6% (84/87; k = 0.920) of cases. The diagnostic accuracy of the two datasets when the presence of any worrisome feature/high-risk stigmata was predictive of malignancy was identical for reader 1 (AUC = 0.622 for both; p = 1.0) and similar for reader 2 (AUC 0.569 and 0.589; p = 0.08) for both MRI datasets.

CONCLUSION

The addition of gadolinium had no significant impact in the diagnosis of a benign versus malignant PCL, with similar intra-observer agreement and diagnostic accuracy for both readers when using contrast-enhanced and unenhanced MRI datasets.

Radiologists and Clinical Trials: Part 2: Practical Statistical Methods for Understanding and Monitoring Independent Reader Performance

Though many clinical trials rely on medical image evaluations for primary or key secondary endpoints, the methods to monitor reader performance are all too often mired in the legacy use of adjudication rates. If misused, this simple metric can be misleading and sometimes entirely contradictory. Furthermore, attempts to overcome the limitations of adjudication rates using de novo or ad hoc methods often ignore well-established research conducted over the last half-century and can lead to inaccurate conclusions or variable interpretations. Underperforming readers can be missed, expert readers retrained, or worse, replaced. This paper aims to standardize reader performance evaluations using proven statistical methods. Additionally, these methods will describe how to discriminate between scenarios of concern and normal medical interpretation variability. Statistical methods are provided for inter-reader and intra-reader variability and bias, including the adjudicator's bias. Finally, we have compiled guidelines for calculating correct sample sizes, considerations for intra-reader memory recall, and applying alternative designs for independent readers.

Staging achilles tendinopathy using ultrasound imaging: the development and investigation of a new ultrasound imaging criteria based on the continuum model of tendon pathology

Aim

To develop a standardised ultrasound imaging (USI)-based criteria for the diagnosis of tendinopathy that aligns with the continuum model of tendon pathology. Secondary aims were to assess both the intra-rater and inter-rater reliability of the criteria.

Methods

A criteria was developed following a face validity assessment and a total of 31 Achilles tendon ultrasound images were analysed. Intra-rater and inter-rater reliability were assessed for overall tendinopathy stage (normal, reactive/early dysrepair or late dysrepair/degenerative) as well as for individual parameters (thickness, echogenicity and vascularity). Quadratic weighted kappa (k_w) was used to report on reliability.

Results

Intra-rater reliability was ‘substantial’ for overall tendinopathy staging (k_w rater A; 0.77, 95% CI 0.59 to 0.94, rater B; 0.70, 95% CI 0.52 to 0.89) and ranged from ‘substantial’ to ‘almost perfect’ for thickness (k_w rater A; 0.75, 95% CI 0.59 to 0.90, rater B; 0.84, 95% CI 0.71 to 0.98), echogenicity (k_w rater A; 0.78, 95% CI 0.62 to 0.95, rater B; 0.73, 95% CI 0.58 to 0.89) and vascularity (k_w rater A; 0.86, 95% CI 0.74 to 0.98, rater B; 0.89, 95% CI 0.79 to 0.99). Inter-rater reliability ranged from ‘substantial’ to ‘almost perfect’ for overall tendinopathy staging (k_w round 1; 0.75, 95% CI 0.58 to 0.91, round 2; 0.81, 95% CI 0.63 to 0.99), thickness (k_w round 1; 0.65, 95% CI 0.48 to 0.83, round 2; 0.77, 95% CI 0.60 to 0.93), echogenicity (k_w round 1; 0.70, 95% CI 0.54 to 0.85, round 2; 0.76, 95% CI 0.58 to 0.94) and vascularity (k_w round 1; 0.89, 95% CI 0.79 to 0.99, round 2; 0.86, 95% CI 0.74 to 0.98). Inter-rater reliability increased from ‘substantial’ in round 1 (k_w 0.75, 95% CI 0.58 to 0.91) to ‘almost perfect’ in round 2 (0.81, 95% CI 0.63 to 0.99).

Conclusion

Intra-rater and inter-rater reliability were ‘substantial’ to ‘almost perfect’ when utilising an USI-based criteria to diagnose Achilles tendinopathy. This is the first study to use the continuum model of tendon pathology to develop an USI-based criteria to diagnose tendinopathy.

Systematic error and cognitive bias in obstetric ultrasound

Linked Comment: Ultrasound Obstet Gynecol 2019; 53: 454-464.

Weekend effect in endovascular stroke treatment: do treatment decisions, procedural times, and outcome depend on time of admission?

Background

Epidemiologic studies identified a ‘weekend effect’ or ‘out-of-hours effect’, which implies that procedural and clinical outcomes of patients with stroke, who are admitted out-of-hours, are less favorable than for patients admitted during working-hours.

Objective

To determine (1) whether our procedural times and clinical outcome were affected by an out-of-hours effect and (2) whether the decision in favor of, or against, endovascular stroke treatment (EST) depends on the time of admission.

Methods

Between February 2010 and January 2015, 6412 consecutive patients presenting with symptoms of acute ischemic stroke were evaluated for EST eligibility according to established local protocols and generally accepted consensus criteria, and dichotomized into working-hours and out-of-hours cohorts according to admission times. Within both groups, patients given EST were identified and the rate of treatment decision, procedural times, and clinical outcome were compared and analyzed.

Results

Clinical and radiological features of patients admitted in working-hours and out-of-hours did not differ significantly. Procedural times and clinical outcome were not affected by an out-of-hours effect (p≥0.054). 221/240 (92.1%) out-of-hours patients and 154/166 (92.8%) working-hours patients who were eligible for EST were transferred to the angiography suite for EST (p=0.798). The rationale not to treat patients who were eligible for EST did not differ between working-hours and out-of-hours admission (p=0.756).

Conclusions

It is possible to produce competitive procedural times regardless of the time of admission and to prevent a treatment decision bias when standard operating procedures are applied consistently.

Do prevalence expectations affect patterns of visual search and decision-making in interpreting CT colonography endoluminal videos?

OBJECTIVE

To assess the effect of expected abnormality prevalence on visual search and decision-making in CT colonography (CTC).

METHODS

13 radiologists interpreted endoluminal CTC fly-throughs of the same group of 10 patient cases, 3 times each. Abnormality prevalence was fixed (50%), but readers were told, before viewing each group, that prevalence was either 20%, 50% or 80% in the population from which cases were drawn. Infrared visual search recording was used. Readers indicated seeing a polyp by clicking a mouse. Multilevel modelling quantified the effect of expected prevalence on outcomes.

RESULTS

Differences between expected prevalence were not statistically significant for time to first pursuit of the polyp (median 0.5 s, each prevalence), pursuit rate when no polyp was on screen (median 2.7 s(-1), each prevalence) or number of mouse clicks [mean 0.75/video (20% prevalence), 0.93 (50%), 0.97 (80%)]. There was weak evidence of increased tendency to look outside the central screen area at 80% prevalence and reduction in positive polyp identifications at 20% prevalence.

CONCLUSION

This study did not find a large effect of prevalence information on most visual search metrics or polyp identification in CTC. Further research is required to quantify effects at lower prevalence and in relation to secondary outcome measures.

ADVANCES IN KNOWLEDGE

Prevalence effects in evaluating CTC have not previously been assessed. In this study, providing expected prevalence information did not have a large effect on diagnostic decisions or patterns of visual search.

Visual memory effects on intraoperator study design: determining a minimum time gap between case reviews to reduce recall bias

OBJECTIVES

The objective of this research was to determine test intervals between intraoperator case reviews to minimize the impact of recall.

METHODS

Three pathologists were presented with a group of 120 slides and subsequently challenged with a study set of 120 slides after 2-week and 4-week intervals. The challenge set consisted of 60 slides seen during the initial review and 60 slides previously unseen within the study. Pathologists rendered a diagnosis for each slide and indicated whether they recalled seeing the slide previously (yes/no).

RESULTS

Two weeks after having been shown 60 cases from a challenge set of 120 cases, the pathologists correctly remembered 26, 22, and 24 cases or 40% overall. After 4 weeks, the pathologists correctly recalled 31% of cases previously seen.

CONCLUSIONS

Pathologists were capable of recalling from memory cases seen previously at 2 and 4 weeks. Recall rates may be sufficiently high to affect intraobserver study design.

[Critical reading of articles about diagnostic tests (part I): Are the results of the study valid?]

In the era of evidence-based medicine, one of the most important skills a radiologist should have is the ability to analyze the diagnostic literature critically. This tutorial aims to present guidelines for determining whether primary diagnostic articles are valid for clinical practice. The following elements should be evaluated: whether the study can be applied to clinical practice, whether the technique was compared to the reference test, whether an appropriate spectrum of patients was included, whether expectation bias and verification bias were limited, the reproducibility of the study, the practical consequences of the study, the confidence intervals for the parameters analyzed, the normal range for continuous variables, and the placement of the test in the context of other diagnostic tests. We use elementary practical examples to illustrate how to select and interpret the literature on diagnostic imaging and specific references to provide more details.

Pragmatism, persistence and patience: a user perspective on strategies for data collection using popular online social networks

The increasing pervasiveness of the internet and social networking globally presents new opportunities and challenges for empirical social science researchers including those in nursing. Developments in computer-mediated communication are not static and there is potential for further advances and innovation in research methods embracing this technology. The aim of this paper is to present a reflexive account and critique of the use of social media as a means of data collection in a study that sought to explore the aesthetics of clinical leadership in contemporary nursing. In doing so, comparisons are drawn from using Twitter, Facebook and e-learning announcements as methods of recruitment and subsequent data collection via an online survey. The pragmatics of the internet and online social networks as vehicles for data collection are discussed. While questions remain about best practice to safeguard the scientific integrity of these approaches and the researchers and research participants who choose to participate, the potential exists for researchers to enhance and expand research methods without compromising rigour and validity. In the interests of sharpening thinking about this means of data collection dialogue and debate are needed on a range of research aspects including but not limited to pragmatics, new requirements in research training and development, legal and ethical guidelines and strengths and limitations encountered.

Expert witness blinding strategies to mitigate bias in radiology malpractice cases: a comprehensive review of the literature

Like all physicians, radiologists in the United States are subject to frequent and costly medical malpractice claims. Legal scholars and physicians concur that the US civil justice system is neither precise nor accurate in determining whether malpractice has truly occurred in cases in which claims are made. Sometimes, this inaccuracy is driven by biases inherent in medical expert-witness opinions. For example, expert-witness testimony involving "missed" radiology findings can be negatively affected by several cognitive biases, such as contextual bias, hindsight bias, and outcome bias. Biases inherent in the US legal system, such as selection bias, compensation bias, and affiliation bias, also play important roles. Fortunately, many of these biases can be significantly mitigated or eliminated through the use of appropriate blinding techniques. This paper reviews the major works on expert-witness blinding in the legal scholarship and the radiology professional literature. Its purpose is to acquaint the reader with the evidence that unblinded expert-witness testimony is tainted by multiple sources of bias and to examine proposed strategies for addressing these biases through blinding.

Methodological quality of diagnostic accuracy studies on non-invasive coronary CT angiography: influence of QUADAS (Quality Assessment of Diagnostic Accuracy Studies included in systematic reviews) items on sensitivity and specificity

OBJECTIVES

To evaluate the methodological quality of diagnostic accuracy studies on coronary computed tomography (CT) angiography using the QUADAS (Quality Assessment of Diagnostic Accuracy Studies included in systematic reviews) tool.

METHODS

Each QUADAS item was individually defined to adapt it to the special requirements of studies on coronary CT angiography. Two independent investigators analysed 118 studies using 12 QUADAS items. Meta-regression and pooled analyses were performed to identify possible effects of methodological quality items on estimates of diagnostic accuracy.

RESULTS

The overall methodological quality of coronary CT studies was merely moderate. They fulfilled a median of 7.5 out of 12 items. Only 9 of the 118 studies fulfilled more than 75 % of possible QUADAS items. One QUADAS item ("Uninterpretable Results") showed a significant influence (P = 0.02) on estimates of diagnostic accuracy with "no fulfilment" increasing specificity from 86 to 90 %. Furthermore, pooled analysis revealed that each QUADAS item that is not fulfilled has the potential to change estimates of diagnostic accuracy.

CONCLUSIONS

The methodological quality of studies investigating the diagnostic accuracy of non-invasive coronary CT is only moderate and was found to affect the sensitivity and specificity. An improvement is highly desirable because good methodology is crucial for adequately assessing imaging technologies.

KEY POINTS

• Good methodological quality is a basic requirement in diagnostic accuracy studies. • Most coronary CT angiography studies have only been of moderate design quality. • Weak methodological quality will affect the sensitivity and specificity. • No improvement in methodological quality was observed over time. • Authors should consider the QUADAS checklist when undertaking accuracy studies.