AI in the Loop: functionalizing fold performance disagreement to monitor automated medical image segmentation workflows

Introduction

Methods that automatically flag poor performing predictions are drastically needed to safely implement machine learning workflows into clinical practice as well as to identify difficult cases during model training.

Methods

Disagreement between the fivefold cross-validation sub-models was quantified using dice scores between folds and summarized as a surrogate for model confidence. The summarized Interfold Dices were compared with thresholds informed by human interobserver values to determine whether final ensemble model performance should be manually reviewed.

Results

The method on all tasks efficiently flagged poor segmented images without consulting a reference standard. Using the median Interfold Dice for comparison, substantial dice score improvements after excluding flagged images was noted for the in-domain CT (0.85 ± 0.20 to 0.91 ± 0.08, 8/50 images flagged) and MR (0.76 ± 0.27 to 0.85 ± 0.09, 8/50 images flagged). Most impressively, there were dramatic dice score improvements in the simulated out-of-distribution task where the model was trained on a radical nephrectomy dataset with different contrast phases predicting a partial nephrectomy all cortico-medullary phase dataset (0.67 ± 0.36 to 0.89 ± 0.10, 122/300 images flagged).

Discussion

Comparing interfold sub-model disagreement against human interobserver values is an effective and efficient way to assess automated predictions when a reference standard is not available. This functionality provides a necessary safeguard to patient care important to safely implement automated medical image segmentation workflows.

Effect of Dataset Size and Medical Image Modality on Convolutional Neural Network Model Performance for Automated Segmentation: A CT and MR Renal Tumor Imaging Study

The aim of this study is to investigate the use of an exponential-plateau model to determine the required training dataset size that yields the maximum medical image segmentation performance. CT and MR images of patients with renal tumors acquired between 1997 and 2017 were retrospectively collected from our nephrectomy registry. Modality-based datasets of 50, 100, 150, 200, 250, and 300 images were assembled to train models with an 80-20 training-validation split evaluated against 50 randomly held out test set images. A third experiment using the KiTS21 dataset was also used to explore the effects of different model architectures. Exponential-plateau models were used to establish the relationship of dataset size to model generalizability performance. For segmenting non-neoplastic kidney regions on CT and MR imaging, our model yielded test Dice score plateaus of [Formula: see text] and [Formula: see text] with the number of training-validation images needed to reach the plateaus of 54 and 122, respectively. For segmenting CT and MR tumor regions, we modeled a test Dice score plateau of [Formula: see text] and [Formula: see text], with 125 and 389 training-validation images needed to reach the plateaus. For the KiTS21 dataset, the best Dice score plateaus for nn-UNet 2D and 3D architectures were [Formula: see text] and [Formula: see text] with number to reach performance plateau of 177 and 440. Our research validates that differing imaging modalities, target structures, and model architectures all affect the amount of training images required to reach a performance plateau. The modeling approach we developed will help future researchers determine for their experiments when additional training-validation images will likely not further improve model performance.

Perioperative and Oncologic Outcomes Associated with Simultaneous Radical Cystectomy and Nephroureterectomy

OBJECTIVE

To compare the perioperative and oncologic outcomes associated with RCNU to a matched cohort undergoing RC alone. Simultaneous radical cystectomy and nephroureterectomy (RCNU) for synchronous upper tract and bladder urothelial carcinoma is an uncommon procedure. Sparse literature exists comparing outcomes in patients treated with radical cystectomy (RC) alone versus RCNU.

METHODS

Adults treated with RCNU for urothelial carcinoma of the bladder (UCB) and upper tract urothelial carcinoma (UTUC) between 1980 and 2020 were identified. Patients were matched 2:1 to patients undergoing RC alone for UCB based on age (+/- 5 years), gender, BMI (+/- 5), Charlson Comorbidity Index, pathologic staging (stage ≤pT2 vs >pT2), and receipt of neoadjuvant chemotherapy. Outcomes included overall survival (OS), recurrence free survival (RFS), cancer specific survival (CSS), 30-day complications, length of stay (LOS), operative time, and estimated blood loss (EBL).

RESULTS

A total of 39 patients undergoing RCNU were identified and matched to 74 patients undergoing RC. There were no significant differences in LOS, EBL, or 30-day complication rates. Operative time was significantly longer in the RC cohort. OS (HR 0.58, CI 0.35-0.97, P = .036) was significantly better for patients undergoing RC alone, while no significant difference was noted in RFS (HR 0.65, 0.34-1.24) and CSS (HR 0.58, CI 0.31-1.08, P = .08).

CONCLUSIONS

Patients undergoing RCNU had significantly lower OS compared to a matched group of patients undergoing RC alone. Perioperative outcomes between the groups did not differ significantly. This data can inform patient counseling for treatment of this rare disease state.

Experience with Telemedicine in a Tertiary Academic Otologic Clinic During the COVID-19 Pandemic

OBJECTIVE

To examine the utility of telemedicine in a tertiary otologic practice.

STUDY DESIGN

Retrospective case series.

SETTING

Tertiary neurotology clinic.

PATIENTS

Consecutive adult patients presenting via video visit between January 2020 and January 2021.

INTERVENTIONS

Televideo modality to conduct visits with patients seeking evaluation for new concerns, second opinions, or routine follow-up for established conditions.

MAIN OUTCOME MEASURES

Success of the televideo visit defined by the televideo visit being sufficient for determining a definitive plan and not requiring deferment of recommendations for a subsequent in-person visit.

RESULTS

A total of 102 televideo visits were performed among 100 unique patients. Of those, 92 (90.2%) visits were for second opinions or evaluation of new concerns, most commonly for vestibular schwannoma (n = 32, 31.4%), followed by sensorineural hearing loss (n = 20, 19.6%). Other visits were conducted for early postoperative follow-up and established general follow-up. In 91.2% of cases (n = 93), patients were successfully evaluated and provided recommendations from the initial video visit. All visits with patients having a diagnosis of meningioma (n = 7), and nearly all with vestibular Schwannoma (97%, n = 31) and sensorineural hearing loss (95%, n = 19) were successful. Of the 79 patients offered surgery as one potential treatment option, 31 patients underwent surgery at our institution by time of review. Patients with unsuccessful visits (n = 9, 8.8%) were advised to schedule additional in-person diagnostic imaging, vestibular testing, or cochlear implant candidacy evaluation to establish a more definitive care plan.

CONCLUSION

Virtual televideo visits were successful for a high percentage of selected patients seen at a tertiary neurotology practice, particularly those seeking evaluation of vestibular schwannoma or sensorineural hearing loss.

Narrative review of male urethral sling for post-prostatectomy stress incontinence: sling type, patient selection, and clinical applications

Male stress urinary incontinence (SUI) following prostate treatment is a devastating complaint for many patients. While the artificial urinary sphincter is the gold standard treatment for male SUI, the urethral sling is also popular due to ease of placement, lack of mechanical complexity, and absence of manual dexterity requirement. A literature review was performed of male urethral sling articles spanning the last zz20 years using the PubMed search engine. Clinical practice guidelines were also reviewed for comparison. Four categories of male urethral sling were evaluated: the transobturator AdVance and AdVance XP, the bone-anchored InVance, the quadratic Virtue, and the adjustable sling series. Well selected patients with mild to moderate urinary incontinence and no prior history of radiation experienced the highest success rates at long-term follow up. Patients with post-prostatectomy climacturia also reported improvement in leakage after sling. Concurrent penile prosthesis and sling techniques were reviewed, with favorable short-term outcomes demonstrated. Male urethral sling is a user-friendly surgical procedure with durable long-term outcomes in carefully selected men with mild stress urinary incontinence. Multiple sling types are available with varying degrees of efficacy and complication rates. Longer follow-up and larger cohort sizes are needed for treatment of newer indications such as climacturia as well as techniques involving dual placement of sling and penile prosthesis.