Visual memory effects on intraoperator study design: determining a minimum time gap between case reviews to reduce recall bias

Best Practice for MRI Diagnostic Accuracy Research With Lessons and Examples from the LI-RADS Individual Participant Data Group

Medical imaging diagnostic test accuracy research is strengthened by adhering to best practices for study design, data collection, data documentation, and study reporting. In this review, key elements of such research are discussed, and specific recommendations provided for optimizing diagnostic accuracy study execution to improve uniformity, minimize common sources of bias and avoid potential pitfalls. Examples are provided regarding study methodology and data collection practices based on insights gained by the liver imaging reporting and data system (LI-RADS) individual participant data group, who have evaluated raw data from numerous MRI diagnostic accuracy studies for risk of bias and data integrity. The goal of this review is to outline strategies for investigators to improve research practices, and to help reviewers and readers better contextualize a study's findings while understanding its limitations. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 3.

Ambient light intensity affecting ultrasound operator detection of liver lesions in cine-clips

INTRODUCTION

Ambient light (AL) is an important factor to improve ultrasound pathology detection. However, there are no established room AL levels recommended during an ultrasound examination. We aim to examine the diagnostic accuracy using different intensity of AL for the detection of liver lesions in anonymised pre-recorded cine-clips.

METHODS

Eight ultrasound operators with 5-14 years' professional experiences were prospectively recruited to evaluate 51 randomised cine-clips directly from one ultrasound machine. These 15-s clips of the right lobe of the liver in longitudinal and transverse planes were meant to simulate the ultrasound examination. Operators reviewed all cine-clips and responded to two questions per cine-clip regarding their detection performance under 3 AL settings; 3, 15 and 25 lux, at one lighting per visit. A repeat visit under each AL was performed to assess for intra-operator variability. Each operator completed six visits in total, with at least a 2-day washout period. The operators' performance was compared against imaging reference standards from contrast CT/MRI for cine-clips with lesion and serial US for those without.

RESULTS

AL with highest degree of diagnostic accuracy was found to be at 25 lux. Results from 8 operators revealed sensitivity ranged from 79% to 100%, specificity ranged from 94% to 100%. Positive and negative predictive values were up to 100% with AL at 25 lux. Both intra-and interrater reliability were excellent at 0.85-1.0 (0.79-0.98) and 0.98 (0.97, 0.99) respectively, with AL at 25 lux.

CONCLUSION

This study proved that ambient light intensity affects the ultrasound operator detection of liver lesions on cine-clips.

IMPLICATIONS FOR PRACTICE

Identifying suitable AL levels will influence future ultrasound room construct.

Reliability and usability of telemedicine evaluations for facial dystonia

This study investigated telemedicine reliability and usability in evaluating facial dystonia grading and treatment complications. Eighty-two telemedicine recordings from 43 adults with blepharospasm (12, 28%) and hemifacial spasm (31, 72%) were obtained (mean age 64.5 ± 9.3 years, 32 females [64%]). Two recorded in-hospital telemedicine visits were arranged with in-person visits at baseline and 4-6 weeks. After 8 weeks, neuro-ophthalmologists who performed the in-person visits re-evaluated the telemedicine video records. Intra-rater agreements in assessing spasm gradings were moderate (severity: kappa = 0.42, 95% confidence interval [CI] 0.21-0.62; frequency: kappa = 0.41, 95% CI 0.21-0.61) with substantial agreement in detecting lagophthalmos (kappa = 0.61, 95% CI 0.36-0.86). Adding symptoms to signs increased sensitivity and negative predictive value (NPV) in detecting lagophthalmos (67%-100% and 94%-100%) and drooping lips (38%-75% and 94%-96%), respectively. Thai version Telehealth Usability Questionnaire showed high mean usability score of 6.5 (SD 0.8) out of 7. Telemedicine could further be developed as an alternative platform to evaluate facial dystonia.

Knowledge of cytology results affects the performance of colposcopy: a crossover study

OBJECTIVE

To determine whether knowledge of cytology affects the colposcopist's diagnostic accuracy in the identification of cervical intraepithelial neoplasia grade 2 and worse (≥ CIN2).

METHOD

In this cross-over study, healthcare professionals interpreted colposcopy images from 80 patient cases with known histological diagnoses. For each case, 2 images taken with a colposcope were provided (native and after acetic acid application). Inclusion criteria consisted of women with a transformation zone type 1 or 2, who had both a cytological and histological diagnosis. Cases were distributed across two online surveys, one including and one omitting the cytology. A wash-out period of six weeks between surveys was implemented. Colposcopists were asked to give their diagnosis for each case as < CIN2 or ≥ CIN2 on both assessments. Statistical analysis was conducted to compare the two interpretations.

RESULTS

Knowledge of cytology significantly improved the sensitivity when interpreting colposcopic images, from 51.1% [95%CI: 39.3 to 62.8] to 63.7% [95%CI: 52.1 to 73.9] and improved the specificity from 63.5% [95%CI: 52.3 to 73.5] to 76.6% [95%CI: 67.2 to 84.0]. Sensitivity was higher by 38.6% when a high-grade cytology (ASC-H, HSIL, AGC) was communicated compared to a low-grade cytology (inflammation, ASC-US, LSIL). Specificity was higher by 31% when a low-grade cytology was communicated compared to a high-grade.

CONCLUSIONS

Our data suggests that knowledge of cytology increases sensitivity and specificity for diagnosis of ≥ CIN2 lesions at colposcopy. Association between cytology and histology may have contributed to the findings.

MarrowQuant 2.0: A Digital Pathology Workflow Assisting Bone Marrow Evaluation in Experimental and Clinical Hematology

Bone marrow (BM) cellularity assessment is a crucial step in the evaluation of BM trephine biopsies for hematologic and nonhematologic disorders. Clinical assessment is based on a semiquantitative visual estimation of the hematopoietic and adipocytic components by hematopathologists, which does not provide quantitative information on other stromal compartments. In this study, we developed and validated MarrowQuant 2.0, an efficient, user-friendly digital hematopathology workflow integrated within QuPath software, which serves as BM quantifier for 5 mutually exclusive compartments (bone, hematopoietic, adipocytic, and interstitial/microvasculature areas and other) and derives the cellularity of human BM trephine biopsies. Instance segmentation of individual adipocytes is realized through the adaptation of the machine-learning-based algorithm StarDist. We calculated BM compartments and adipocyte size distributions of hematoxylin and eosin images obtained from 250 bone specimens, from control subjects and patients with acute myeloid leukemia or myelodysplastic syndrome, at diagnosis and follow-up, and measured the agreement of cellularity estimates by MarrowQuant 2.0 against visual scores from 4 hematopathologists. The algorithm was capable of robust BM compartment segmentation with an average mask accuracy of 86%, maximal for bone (99%), hematopoietic (92%), and adipocyte (98%) areas. MarrowQuant 2.0 cellularity score and hematopathologist estimations were highly correlated (R² = 0.92-0.98, intraclass correlation coefficient [ICC] = 0.98; interobserver ICC = 0.96). BM compartment segmentation quantitatively confirmed the reciprocity of the hematopoietic and adipocytic compartments. MarrowQuant 2.0 performance was additionally tested for cellularity assessment of specimens prospectively collected from clinical routine diagnosis. After special consideration for the choice of the cellularity equation in specimens with expanded stroma, performance was similar in this setting (R² = 0.86, n = 42). Thus, we conclude that these validation experiments establish MarrowQuant 2.0 as a reliable tool for BM cellularity assessment. We expect this workflow will serve as a clinical research tool to explore novel biomarkers related to BM stromal components and may contribute to further validation of future digitalized diagnostic hematopathology workstreams.

Relevance of the College of American Pathologists guideline for validating whole slide imaging for diagnostic purposes to cytopathology

Whole slide imaging (WSI) allows pathologists to view virtual versions of slides on computer monitors. With increasing adoption of digital pathology, laboratories have begun to validate their WSI systems for diagnostic purposes according to reference guidelines. Among these the College of American Pathologists (CAP) guideline includes three strong recommendations (SRs) and nine good practice statements (GPSs). To date, the application of WSI to cytopathology has been beyond the scope of the CAP guideline due to limited evidence. Herein we systematically reviewed the published literature on WSI validation studies in cytology. A systematic search was carried out in PubMed-MEDLINE and Embase databases up to November 2021 to identify all publications regarding validation of WSI in cytology. Each article was reviewed to determine if SRs and/or GPSs recommended by the CAP guideline were adequately satisfied. Of 3963 retrieved articles, 25 were included. Only 4/25 studies (16%) satisfied all three SRs, with only one publication (1/25, 4%) fulfilling all three SRs and nine GPSs. Lack of a suitable validation dataset was the main missing SR (16/25, 64%) and less than a third of the studies reported intra-observer variability data (7/25, 28%). Whilst the CAP guideline for WSI validation in clinical practice helped the widespread adoption of digital pathology, more evidence is required to routinely employ WSI for diagnostic purposes in cytopathology practice. More dedicated validation studies satisfying all SRs and/or GPSs recommended by the CAP are needed to help expedite the use of WSI for primary diagnosis in cytopathology.

Validating Whole Slide Imaging Systems for Diagnostic Purposes in Pathology

CONTEXT.—

The original guideline, "Validating Whole Slide Imaging for Diagnostic Purposes in Pathology," was published in 2013 and included 12 guideline statements. The College of American Pathologists convened an expert panel to update the guideline following standards established by the National Academies of Medicine for developing trustworthy clinical practice guidelines.

OBJECTIVE.—

To assess evidence published since the release of the original guideline and provide updated recommendations for validating whole slide imaging (WSI) systems used for diagnostic purposes.

DESIGN.—

An expert panel performed a systematic review of the literature. Frozen sections, anatomic pathology specimens (biopsies, curettings, and resections), and hematopathology cases were included. Cytology cases were excluded. Using the Grading of Recommendations Assessment, Development, and Evaluation approach, the panel reassessed and updated the original guideline recommendations.

RESULTS.—

Three strong recommendations and 9 good practice statements are offered to assist laboratories with validating WSI digital pathology systems.

CONCLUSIONS.—

Systematic review of literature following release of the 2013 guideline reaffirms the use of a validation set of at least 60 cases, establishing intraobserver diagnostic concordance between WSI and glass slides and the use of a 2-week washout period between modalities. Although all discordances between WSI and glass slide diagnoses discovered during validation need to be reconciled, laboratories should be particularly concerned if their overall WSI-glass slide concordance is less than 95%.

Validation of digital microscopy: Review of validation methods and sources of bias

Digital microscopy (DM) is increasingly replacing traditional light microscopy (LM) for performing routine diagnostic and research work in human and veterinary pathology. The DM workflow encompasses specimen preparation, whole-slide image acquisition, slide retrieval, and the workstation, each of which has the potential (depending on the technical parameters) to introduce limitations and artifacts into microscopic examination by pathologists. Performing validation studies according to guidelines established in human pathology ensures that the best-practice approaches for patient care are not deteriorated by implementing DM. Whereas current publications on validation studies suggest an overall high reliability of DM, each laboratory is encouraged to perform an individual validation study to ensure that the DM workflow performs as expected in the respective clinical or research environment. With the exception of validation guidelines developed by the College of American Pathologists in 2013 and its update in 2021, there is no current review of the application of methods fundamental to validation. We highlight that there is high methodological variation between published validation studies, each having advantages and limitations. The diagnostic concordance rate between DM and LM is the most relevant outcome measure, which is influenced (regardless of the viewing modality used) by different sources of bias including complexity of the cases examined, diagnostic experience of the study pathologists, and case recall. Here, we review 3 general study designs used for previous publications on DM validation as well as different approaches for avoiding bias.

Artificial Intelligence-Based Screening for Mycobacteria in Whole-Slide Images of Tissue Samples

OBJECTIVES

This study aimed to develop and validate a deep learning algorithm to screen digitized acid fast-stained (AFS) slides for mycobacteria within tissue sections.

METHODS

A total of 441 whole-slide images (WSIs) of AFS tissue material were used to develop a deep learning algorithm. Regions of interest with possible acid-fast bacilli (AFBs) were displayed in a web-based gallery format alongside corresponding WSIs for pathologist review. Artificial intelligence (AI)-assisted analysis of another 138 AFS slides was compared to manual light microscopy and WSI evaluation without AI support.

RESULTS

Algorithm performance showed an area under the curve of 0.960 at the image patch level. More AI-assisted reviews identified AFBs than manual microscopy or WSI examination (P < .001). Sensitivity, negative predictive value, and accuracy were highest for AI-assisted reviews. AI-assisted reviews also had the highest rate of matching the original sign-out diagnosis, were less time-consuming, and were much easier for pathologists to perform (P < .001).

CONCLUSIONS

This study reports the successful development and clinical validation of an AI-based digital pathology system to screen for AFBs in anatomic pathology material. AI assistance proved to be more sensitive and accurate, took pathologists less time to screen cases, and was easier to use than either manual microscopy or viewing WSIs.

Comparison of reporting radiographers' and medical doctors' performance in reporting radiographs of the appendicular skeleton, referred by the emergency department

INTRODUCTION

There is an increasing trend towards deploying reporting radiographers in Danish hospitals who, among various professional groups, interpret and report skeletal radiographs from the emergency department (ED). This study aimed to compare the quality of the reports issued by reporting radiographers to three different groups of medical doctors (MDs) who interpret or report skeletal radiographs at the ED.

METHODS

Four professional groups (i.e. four reporting radiographers, two radiology trainees, two orthopaedic senior trainees, and two orthopaedic trainees) reported 100 radiographs of the appendicular skeleton. The Consequence of clinical Outcome score (CO-score), accuracy, sensitivity, and specificity of each group were compared. The relative risk of a false-negative, false-positive or wrong result, the risk of a serious error, as well as the odds ratio of a more severe CO-score for each of the three MD groups, were compared to the reporting radiographers.

RESULTS

There was a significant difference between the groups in reference to the CO-score (P ≤ 0.001), accuracy (P = .003), specificity (P = .022), and in the proportion of serious errors (P ≤ 0.001). Compared to the reporting radiographers, all three groups of MDs showed a significantly higher CO-score and a significantly increased risk of a wrong result. Moreover, two of the MD groups showed a significantly increased risk of a false-positive result and for severe errors.

CONCLUSION

Based on the CO-score, the relative risk of errors, which could potentially cause malpractice in treatment and patient recall, significantly decreased when the reports were completed by reporting radiographers.

IMPLICATIONS FOR PRACTICE

To explore the need for a 24-h radiographer reporting service to the EDs, an upscaled study, like the current, with more participants representing the professional groups is highly recommended.

Immediate effects of lower limb loading exercise during stepping with and without augmented loading feedback on mobility of ambulatory individuals with spinal cord injury: a single-blinded, randomized, cross-over trial

STUDY DESIGN

Single-blinded, randomized, cross-over design.

OBJECTIVES

To compare the immediate effects of bodyweight shifting and lower limb loading (LLL) exercise during stepping with and without augmented loading feedback, followed by overground walking, on the mobility of ambulatory individuals with spinal cord injury (SCI).

SETTING

Academic laboratory center.

METHODS

Thirty participants with SCI were trained using a single intervention session consisting of repetitive bodyweight shifting and LLL exercises during stepping with or without external feedback (10 min/leg) followed by overground walking (10 min) with a 2-week washout period, in a random sequence. The timed up-and-go test (TUG) (primary outcome), 10-m walk test (10MWT), five times sit-to-stand test (FTSST), and maximal LLL were measured 1 day before and immediately after each training session.

RESULTS

Significant improvement was found following both training sessions, excepting the TUG and LLL of the less-affected leg, where improvement was found only after training using augmented feedback. Moreover, the improvement following the training with feedback was significantly greater than that after training without feedback. The mean (95% CI) between-group differences for the TUG = 1.9 [0.6-3.3]s, 10MWT = 0.1 [0.0-0.1]m/s, FTSST = 1.0 [1.5-4.8]s, LLL = 3.1 [1.5-4.8]-2.8 [0.8-4.9]%bodyweight, p < 0.05.

CONCLUSIONS

The training programs immediately enhanced the mobility of ambulatory individuals with chronic SCI (post-injury time >6 years), particularly the training with augmented loading feedback. The findings offer another effective rehabilitation strategy that can be applied in various clinical and home-based settings.

Reliability and Variability Assessment of Femoral Artery Pseudoaneurysm Measurements Between Pre- and Postprocessed B-mode Ultrasound Images

Objective:

To assess the reliability and variability of femoral artery pseudoaneurysm (FAP) measurements between pre- and postprocessed sonograms acquired in a major medical center in Saudi Arabia as compared with results obtained from a major medical center in the United States.

Methods:

Retrospective image analysis was conducted on 23 FAP sonograms, which were evaluated by four observers. Observers measured FAP sac and neck from pre- and postprocessed images and remeasured again after 2 weeks, to avoid recall bias.

Results:

The use of image processing was more profound for the novice observers in measuring FAP neck width and length. The intraclass correlation coefficient (ICC) for FAP neck width improved after segmentation from 0.63 to 0.91; in contrast, the ICC improved from 0.91 to 0.97 for experts. The average ICCs for FAP neck length improved from 0.40 to 0.79 for novices and from 0.86 to 0.95 for experts. The largest variation of values, within observers, were for neck length obtained from the original images. The range varied from 0.16 to 0.37 cm and was reduced to 0.10 to 0.18 cm with segmented images.

Conclusion:

As demonstrated previously, sonographic image processing resulted in increased reliability and decreased variability for FAP measurements.

Intra- and Interobserver Reliability and Variability of Femoral Artery Pseudoaneurysm Measurements Between Pre- and Postprocessed B-mode Sonographic Images

The objective was to enhance the contrast and segment the femoral artery pseudoaneurysm (FAP) area from patients’ ultrasonography (US) images. In addition, this study compared the intra- and interobserver reliability and variability of the FAP measurements from the original, contrast-enhanced, and segmented US. Retrospectively, 25 FAP sonograms were evaluated by four observers (two trained and two novice). They twice measured the FAP body width, neck width, and length from original, enhanced, and segmented US. The intra- and interobserver reliability for measuring FAP body width, neck width, and neck length improved by 10% after enhancing the original 2D US contrast and segmenting the region of interest across all observers. Also, the intra- and interobserver variability among measurements across all observers decreased by 44%. Using US processing was more profound for novice observers (intraclass correlation coefficient [ICC], 0.76–0.93) compared to trained observers (ICC, 0.94–0.99). US postprocessing resulted in a decrease in variability for FAP measurements.

An update on the validation of whole slide imaging systems following FDA approval of a system for a routine pathology diagnostic service in the United States

Pathologists have used light microscopes and glass slides to interpret the histologic appearance of normal and diseased tissues for more than 150 years. The quality of both microtomes used to cut tissue sections and microscopes has improved significantly during the past few decades, but the process of rendering diagnoses has changed little. By contrast, major advances in digital technology have occurred since the introduction of hand held electronic devices, including the development of whole slide imaging (WSI) systems with software packages that can convert microscope images into virtual (digital) slides that can be viewed on computer monitors and via the internet. To date, however, these technological developments have had minimal impact on the way pathologists perform their daily work, with the exception of using computers to access electronic medical records and scholarly web sites for pertinent information to assist interpretation of cases. Traditional practice is likely to change significantly during the next decade, especially since the Federal Drug Administration in the USA has approved the first WSI system for routine diagnostic practice. I review here the development and slow acceptance of WSI by pathology departments. I focus on recent advances in validation of WSI systems that is required for routine diagnostic reporting of pathology cases using this technology.

The Diagnostic Concordance of Whole Slide Imaging and Light Microscopy: A Systematic Review

CONTEXT

-Light microscopy (LM) is considered the reference standard for diagnosis in pathology. Whole slide imaging (WSI) generates digital images of cellular and tissue samples and offers multiple advantages compared with LM. Currently, WSI is not widely used for primary diagnosis. The lack of evidence regarding concordance between diagnoses rendered by WSI and LM is a significant barrier to both regulatory approval and uptake.

OBJECTIVE

-To examine the published literature on the concordance of pathologic diagnoses rendered by WSI compared with those rendered by LM.

DATA SOURCES

-We conducted a systematic review of studies assessing the concordance of pathologic diagnoses rendered by WSI and LM. Studies were identified following a systematic search of Medline (Medline Industries, Mundelein, Illinois), Medline in progress (Medline Industries), EMBASE (Elsevier, Amsterdam, the Netherlands), and the Cochrane Library (Wiley, London, England), between 1999 and March 2015.

CONCLUSIONS

-Thirty-eight studies were included in the review. The mean diagnostic concordance of WSI and LM, weighted by the number of cases per study, was 92.4%. The weighted mean κ coefficient between WSI and LM was 0.75, signifying substantial agreement. Of the 30 studies quoting percentage concordance, 18 (60%) showed a concordance of 90% or greater, of which 10 (33%) showed a concordance of 95% or greater. This review found evidence to support a high level of diagnostic concordance. However, there were few studies, many were small, and they varied in quality, suggesting that further validation studies are still needed.