1
|
Deckert A, Anders S, Morales I, De Allegri M, Nguyen HT, Souares A, McMahon S, Meurer M, Burk R, Lou D, Brugnara L, Sand M, Koeppel L, Maier-Hein L, Ross T, Adler TJ, Brenner S, Dyer C, Herbst K, Ovchinnikova S, Marx M, Schnitzler P, Knop M, Bärnighausen T, Denkinger CM. Correction: Comparison of Four Active SARS-CoV-2 Surveillance Strategies in Representative Population Sample Points: Two-Factor Factorial Randomized Controlled Trial. JMIR Public Health Surveill 2024; 10:e57203. [PMID: 38364221 PMCID: PMC10907930 DOI: 10.2196/57203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Accepted: 02/07/2024] [Indexed: 02/18/2024] Open
Abstract
[This corrects the article DOI: 10.2196/44204.].
Collapse
Affiliation(s)
| | - Simon Anders
- Center for Molecular Biology HeidelbergHeidelbergGermany
| | - Ivonne Morales
- Division of Infectious Disease and Tropical Medicine, Heidelberg University HospitalHeidelbergGermany
| | | | | | | | | | | | - Robin Burk
- Center for Molecular Biology HeidelbergHeidelbergGermany
| | - Dan Lou
- Center for Molecular Biology HeidelbergHeidelbergGermany
| | - Lucia Brugnara
- evaplan GmbH at the University HospitalHeidelbergGermany
| | - Matthias Sand
- GESIS Leibniz-Institute for the Social SciencesMannheimGermany
| | - Lisa Koeppel
- Division of Infectious Disease and Tropical Medicine, Heidelberg University HospitalHeidelbergGermany
| | - Lena Maier-Hein
- Division of Computer Assisted Medical Interventions, German Cancer Research CentreHeidelbergGermany
| | - Tobias Ross
- Division of Computer Assisted Medical Interventions, German Cancer Research CentreHeidelbergGermany
| | - Tim J Adler
- Division of Computer Assisted Medical Interventions, German Cancer Research CentreHeidelbergGermany
| | | | | | - Konrad Herbst
- Center for Molecular Biology HeidelbergHeidelbergGermany
| | | | - Michael Marx
- evaplan GmbH at the University HospitalHeidelbergGermany
| | - Paul Schnitzler
- Center of Infectious Diseases, Virology, Heidelberg University HospitalHeidelbergGermany
| | - Michael Knop
- Center for Molecular Biology HeidelbergHeidelbergGermany
| | | | - Claudia M Denkinger
- Division of Infectious Disease and Tropical Medicine, Heidelberg University HospitalHeidelbergGermany
| |
Collapse
|
2
|
Deckert A, Anders S, Morales I, De Allegri M, Nguyen HT, Souares A, McMahon S, Meurer M, Burk R, Lou D, Brugnara L, Sand M, Koeppel L, Maier-Hein L, Ross T, Adler TJ, Brenner S, Dyer C, Herbst K, Ovchinnikova S, Marx M, Schnitzler P, Knop M, Bärnighausen T, Denkinger CM. Comparison of Four Active SARS-CoV-2 Surveillance Strategies in Representative Population Sample Points: Two-Factor Factorial Randomized Controlled Trial. JMIR Public Health Surveill 2023; 9:e44204. [PMID: 37235704 PMCID: PMC10437130 DOI: 10.2196/44204] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2022] [Revised: 03/30/2023] [Accepted: 05/24/2023] [Indexed: 05/28/2023] Open
Abstract
BACKGROUND The COVID-19 pandemic is characterized by rapid increases in infection burden owing to the emergence of new variants with higher transmissibility and immune escape. To date, monitoring the COVID-19 pandemic has mainly relied on passive surveillance, yielding biased epidemiological measures owing to the disproportionate number of undetected asymptomatic cases. Active surveillance could provide accurate estimates of the true prevalence to forecast the evolution of the pandemic, enabling evidence-based decision-making. OBJECTIVE This study compared 4 different approaches of active SARS-CoV-2 surveillance focusing on feasibility and epidemiological outcomes. METHODS A 2-factor factorial randomized controlled trial was conducted in 2020 in a German district with 700,000 inhabitants. The epidemiological outcome comprised SARS-CoV-2 prevalence and its precision. The 4 study arms combined 2 factors: individuals versus households and direct testing versus testing conditioned on symptom prescreening. Individuals aged ≥7 years were eligible. Altogether, 27,908 addresses from 51 municipalities were randomly allocated to the arms and 15 consecutive recruitment weekdays. Data collection and logistics were highly digitized, and a website in 5 languages enabled low-barrier registration and tracking of results. Gargle sample collection kits were sent by post. Participants collected a gargle sample at home and mailed it to the laboratory. Samples were analyzed with reverse transcription loop-mediated isothermal amplification (RT-LAMP); positive and weak results were confirmed with real-time reverse transcription-polymerase chain reaction (RT-PCR). RESULTS Recruitment was conducted between November 18 and December 11, 2020. The response rates in the 4 arms varied between 34.31% (2340/6821) and 41.17% (2043/4962). The prescreening classified 16.61% (1207/7266) of the patients as COVID-19 symptomatic. Altogether, 4232 persons without prescreening and 7623 participating in the prescreening provided 5351 gargle samples, of which 5319 (99.4%) could be analyzed. This yielded 17 confirmed SARS-CoV-2 infections and a combined prevalence of 0.36% (95% CI 0.14%-0.59%) in the arms without prescreening and 0.05% (95% CI 0.00%-0.108%) in the arms with prescreening (initial contacts only). Specifically, we found a prevalence of 0.31% (95% CI 0.06%-0.58%) for individuals and 0.35% (95% CI 0.09%-0.61%) for households, and lower estimates with prescreening (0.07%, 95% CI 0.0%-0.15% for individuals and 0.02%, 95% CI 0.0%-0.06% for households). Asymptomatic infections occurred in 27% (3/11) of the positive cases with symptom data. The 2 arms without prescreening performed the best regarding effectiveness and accuracy. CONCLUSIONS This study showed that postal mailing of gargle sample kits and returning home-based self-collected liquid gargle samples followed by high-sensitivity RT-LAMP analysis is a feasible way to conduct active SARS-CoV-2 population surveillance without burdening routine diagnostic testing. Efforts to improve participation rates and integration into the public health system may increase the potential to monitor the course of the pandemic. TRIAL REGISTRATION Deutsches Register Klinischer Studien (DRKS) DRKS00023271; https://tinyurl.com/3xenz68a. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) RR2-10.1186/s13063-021-05619-5.
Collapse
Affiliation(s)
| | - Simon Anders
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | - Ivonne Morales
- Division of Infectious Disease and Tropical Medicine, Heidelberg University Hospital, Heidelberg, Germany
| | | | - Hoa Thi Nguyen
- Heidelberg Institute of Global Health, Heidelberg, Germany
| | | | | | - Matthias Meurer
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | - Robin Burk
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | - Dan Lou
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | - Lucia Brugnara
- evaplan GmbH at the University Hospital, Heidelberg, Germany
| | - Matthias Sand
- GESIS Leibniz-Institute for the Social Sciences, Mannheim, Germany
| | - Lisa Koeppel
- Division of Infectious Disease and Tropical Medicine, Heidelberg University Hospital, Heidelberg, Germany
| | - Lena Maier-Hein
- Division of Computer Assisted Medical Interventions, German Cancer Research Centre, Heidelberg, Germany
| | - Tobias Ross
- Division of Computer Assisted Medical Interventions, German Cancer Research Centre, Heidelberg, Germany
| | - Tim J Adler
- Division of Computer Assisted Medical Interventions, German Cancer Research Centre, Heidelberg, Germany
| | | | | | - Konrad Herbst
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | | | - Michael Marx
- evaplan GmbH at the University Hospital, Heidelberg, Germany
| | - Paul Schnitzler
- Center of Infectious Diseases, Virology, Heidelberg University Hospital, Heidelberg, Germany
| | - Michael Knop
- Center for Molecular Biology Heidelberg, Heidelberg, Germany
| | | | - Claudia M Denkinger
- Division of Infectious Disease and Tropical Medicine, Heidelberg University Hospital, Heidelberg, Germany
| |
Collapse
|
3
|
Tran TN, Adler TJ, Yamlahi A, Christodoulou E, Godau P, Reinke A, Tizabi MD, Sauer P, Persicke T, Albert JG, Maier-Hein L. Sources of performance variability in deep learning-based polyp detection. Int J Comput Assist Radiol Surg 2023:10.1007/s11548-023-02936-9. [PMID: 37266886 PMCID: PMC10329574 DOI: 10.1007/s11548-023-02936-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Accepted: 04/24/2023] [Indexed: 06/03/2023]
Abstract
PURPOSE Validation metrics are a key prerequisite for the reliable tracking of scientific progress and for deciding on the potential clinical translation of methods. While recent initiatives aim to develop comprehensive theoretical frameworks for understanding metric-related pitfalls in image analysis problems, there is a lack of experimental evidence on the concrete effects of common and rare pitfalls on specific applications. We address this gap in the literature in the context of colon cancer screening. METHODS Our contribution is twofold. Firstly, we present the winning solution of the Endoscopy Computer Vision Challenge on colon cancer detection, conducted in conjunction with the IEEE International Symposium on Biomedical Imaging 2022. Secondly, we demonstrate the sensitivity of commonly used metrics to a range of hyperparameters as well as the consequences of poor metric choices. RESULTS Based on comprehensive validation studies performed with patient data from six clinical centers, we found all commonly applied object detection metrics to be subject to high inter-center variability. Furthermore, our results clearly demonstrate that the adaptation of standard hyperparameters used in the computer vision community does not generally lead to the clinically most plausible results. Finally, we present localization criteria that correspond well to clinical relevance. CONCLUSION We conclude from our study that (1) performance results in polyp detection are highly sensitive to various design choices, (2) common metric configurations do not reflect the clinical need and rely on suboptimal hyperparameters and (3) comparison of performance across datasets can be largely misleading. Our work could be a first step towards reconsidering common validation strategies in deep learning-based colonoscopy and beyond.
Collapse
Affiliation(s)
- T N Tran
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany.
| | - T J Adler
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
| | - A Yamlahi
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
| | - E Christodoulou
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
| | - P Godau
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, Germany
| | - A Reinke
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
| | - M D Tizabi
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
| | - P Sauer
- Interdisciplinary Endoscopy Center (IEZ), University Hospital Heidelberg, Heidelberg, Germany
| | - T Persicke
- Department of Gastroenterology, Hepatology and Endocrinology, Robert-Bosch Hospital (RBK), Stuttgart, Germany
| | - J G Albert
- Department of Gastroenterology, Hepatology and Endocrinology, Robert-Bosch Hospital (RBK), Stuttgart, Germany
- Clinic for General Internal Medicine, Gastroenterology, Hepatology and Infectiology, Pneumology, Klinikum Stuttgart, Stuttgart, Germany
| | - L Maier-Hein
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, Germany
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a Partnership Between DKFZ and University Medical Center Heidelberg, Heidelberg, Germany
| |
Collapse
|
4
|
Ayala L, Adler TJ, Seidlitz S, Wirkert S, Engels C, Seitel A, Sellner J, Aksenov A, Bodenbach M, Bader P, Baron S, Vemuri A, Wiesenfarth M, Schreck N, Mindroc D, Tizabi M, Pirmann S, Everitt B, Kopp-Schneider A, Teber D, Maier-Hein L. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Sci Adv 2023; 9:eadd6778. [PMID: 36897951 PMCID: PMC10005169 DOI: 10.1126/sciadv.add6778] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/27/2022] [Accepted: 02/03/2023] [Indexed: 06/18/2023]
Abstract
Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.
Collapse
Affiliation(s)
- Leonardo Ayala
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Medical Faculty, Heidelberg University, Heidelberg, Germany
| | - Tim J. Adler
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Silvia Seidlitz
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany
| | - Sebastian Wirkert
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Alexander Seitel
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jan Sellner
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany
| | | | | | - Pia Bader
- Städtisches Klinikum Karlsruhe, Karlsruhe, Germany
| | | | - Anant Vemuri
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Manuel Wiesenfarth
- Division of Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Nicholas Schreck
- Division of Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Diana Mindroc
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Minu Tizabi
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sebastian Pirmann
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Brittaney Everitt
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Dogu Teber
- Städtisches Klinikum Karlsruhe, Karlsruhe, Germany
| | - Lena Maier-Hein
- Division of Intelligent Medical Systems, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Medical Faculty, Heidelberg University, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany
| |
Collapse
|
5
|
Roß T, Bruno P, Reinke A, Wiesenfarth M, Koeppel L, Full PM, Pekdemir B, Godau P, Trofimova D, Isensee F, Adler TJ, Tran TN, Moccia S, Calimeri F, Müller-Stich BP, Kopp-Schneider A, Maier-Hein L. Beyond rankings: Learning (more) from algorithm validation. Med Image Anal 2023; 86:102765. [PMID: 36965252 DOI: 10.1016/j.media.2023.102765] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2021] [Revised: 05/24/2022] [Accepted: 02/08/2023] [Indexed: 03/06/2023]
Abstract
Challenges have become the state-of-the-art approach to benchmark image analysis algorithms in a comparative manner. While the validation on identical data sets was a great step forward, results analysis is often restricted to pure ranking tables, leaving relevant questions unanswered. Specifically, little effort has been put into the systematic investigation on what characterizes images in which state-of-the-art algorithms fail. To address this gap in the literature, we (1) present a statistical framework for learning from challenges and (2) instantiate it for the specific task of instrument instance segmentation in laparoscopic videos. Our framework relies on the semantic meta data annotation of images, which serves as foundation for a General Linear Mixed Models (GLMM) analysis. Based on 51,542 meta data annotations performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Segmentation Challenge (ROBUST-MIS) challenge 2019 and revealed underexposure, motion and occlusion of instruments as well as the presence of smoke or other objects in the background as major sources of algorithm failure. Our subsequent method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail. Due to the objectivity and generic applicability of our approach, it could become a valuable tool for validation in the field of medical image analysis and beyond.
Collapse
Affiliation(s)
- Tobias Roß
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; Medical Faculty, Heidelberg University, Heidelberg, Germany; Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany.
| | - Pierangela Bruno
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; Department of Mathematics and Computer Science, University of Calabria, Rende, Italy
| | - Annika Reinke
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Mathematics and Computer Science, Heidelberg University, Germany
| | - Manuel Wiesenfarth
- Division of Biostatistics, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Lisa Koeppel
- Section Clinical Tropical Medicine, Heidelberg University, Heidelberg, Germany
| | - Peter M Full
- Medical Faculty, Heidelberg University, Heidelberg, Germany; Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Bünyamin Pekdemir
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Patrick Godau
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Mathematics and Computer Science, Heidelberg University, Germany
| | - Darya Trofimova
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; HIP Applied Computer Vision Lab, MIC, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Fabian Isensee
- Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), Heidelberg, Germany; HIP Applied Computer Vision Lab, MIC, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tim J Adler
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Thuy N Tran
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sara Moccia
- The BioRobotics Institute and Department of Excellence in Robotics and AI, Scuola Superiore Sant'Anna, Italy
| | - Francesco Calimeri
- Department of Mathematics and Computer Science, University of Calabria, Rende, Italy
| | - Beat P Müller-Stich
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany
| | | | - Lena Maier-Hein
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany; Medical Faculty, Heidelberg University, Heidelberg, Germany; Helmholtz Imaging, German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Mathematics and Computer Science, Heidelberg University, Germany; Germany and National Center for Tumor Diseases (NCT), Heidelberg, Germany
| |
Collapse
|
6
|
Müller LR, Petersen J, Yamlahi A, Wise P, Adler TJ, Seitel A, Kowalewski KF, Müller B, Kenngott H, Nickel F, Maier-Hein L. Robust hand tracking for surgical telestration. Int J Comput Assist Radiol Surg 2022; 17:1477-1486. [PMID: 35624404 PMCID: PMC9307534 DOI: 10.1007/s11548-022-02637-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2022] [Accepted: 04/06/2022] [Indexed: 11/30/2022]
Abstract
PURPOSE As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs. METHODS We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching. RESULTS According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied. CONCLUSION Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.
Collapse
Affiliation(s)
- Lucas-Raphael Müller
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
| | - Jens Petersen
- Division of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Amine Yamlahi
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Philipp Wise
- Department for General, Visceral and Transplantation Surgery, Mannheim University Hospital, Heidelberg, Germany
| | - Tim J Adler
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Alexander Seitel
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Karl-Friedrich Kowalewski
- Department of Urology and Urosurgery, Medical Faculty Mannheim, Heidelberg University Hospital, Heidelberg, Germany
| | - Beat Müller
- Department for General, Visceral and Transplantation Surgery, Mannheim University Hospital, Heidelberg, Germany
| | - Hannes Kenngott
- Department for General, Visceral and Transplantation Surgery, Mannheim University Hospital, Heidelberg, Germany
| | - Felix Nickel
- Department for General, Visceral and Transplantation Surgery, Mannheim University Hospital, Heidelberg, Germany.
| | - Lena Maier-Hein
- Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
- Medical Faculty, Heidelberg University, Heidelberg, Germany
| |
Collapse
|
7
|
Seidlitz S, Sellner J, Odenthal J, Özdemir B, Studier-Fischer A, Knödler S, Ayala L, Adler TJ, Kenngott HG, Tizabi M, Wagner M, Nickel F, Müller-Stich BP, Maier-Hein L. Robust deep learning-based semantic organ segmentation in hyperspectral images. Med Image Anal 2022; 80:102488. [DOI: 10.1016/j.media.2022.102488] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2021] [Revised: 03/28/2022] [Accepted: 05/20/2022] [Indexed: 12/15/2022]
|
8
|
Maier-Hein L, Wagner M, Ross T, Reinke A, Bodenstedt S, Full PM, Hempe H, Mindroc-Filimon D, Scholz P, Tran TN, Bruno P, Kisilenko A, Müller B, Davitashvili T, Capek M, Tizabi MD, Eisenmann M, Adler TJ, Gröhl J, Schellenberg M, Seidlitz S, Lai TYE, Pekdemir B, Roethlingshoefer V, Both F, Bittel S, Mengler M, Mündermann L, Apitz M, Kopp-Schneider A, Speidel S, Nickel F, Probst P, Kenngott HG, Müller-Stich BP. Heidelberg colorectal data set for surgical data science in the sensor operating room. Sci Data 2021; 8:101. [PMID: 33846356 PMCID: PMC8042116 DOI: 10.1038/s41597-021-00882-2] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2020] [Accepted: 02/24/2021] [Indexed: 11/30/2022] Open
Abstract
Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.
Collapse
Affiliation(s)
- Lena Maier-Hein
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany.
| | - Martin Wagner
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Tobias Ross
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
- Heidelberg University, Seminarstraße 2, 69117, Heidelberg, Germany
| | - Annika Reinke
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
- Heidelberg University, Seminarstraße 2, 69117, Heidelberg, Germany
| | - Sebastian Bodenstedt
- Division of Translational Surgical Oncology, National Center for Tumor Diseases, Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
| | - Peter M Full
- Heidelberg University, Seminarstraße 2, 69117, Heidelberg, Germany
- Division of Medical Image Computing (MIC), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Hellena Hempe
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Diana Mindroc-Filimon
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Patrick Scholz
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
- HIDSS4Health - Helmholtz Information and Data Science School for Health, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Thuy Nuong Tran
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Pierangela Bruno
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
- Department of Mathematics and Computer Science, University of Calabria, Via Pietro Bucci, 87036 Arcavacata, Rende, CS, Italy
| | - Anna Kisilenko
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Benjamin Müller
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Tornike Davitashvili
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Manuela Capek
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Minu D Tizabi
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Matthias Eisenmann
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Tim J Adler
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Janek Gröhl
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Melanie Schellenberg
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Silvia Seidlitz
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
- HIDSS4Health - Helmholtz Information and Data Science School for Health, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - T Y Emmy Lai
- Division of Medical Image Computing (MIC), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Bünyamin Pekdemir
- Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | | | - Fabian Both
- understandAI GmbH, An der RaumFabrik 34, 76227, Karlsruhe, Germany
- International Max Planck Research School for Intelligent Systems Tuebingen, University of Tuebingen, Geschwister-Scholl-Platz, 72074, Tübingen, Germany
| | - Sebastian Bittel
- understandAI GmbH, An der RaumFabrik 34, 76227, Karlsruhe, Germany
- BMW Group, Heidemannstraße 164, 80939, Munich, Germany
| | - Marc Mengler
- understandAI GmbH, An der RaumFabrik 34, 76227, Karlsruhe, Germany
| | - Lars Mündermann
- Corporate Research & Technology, Data-Assisted Solutions, KARL STORZ SE & Co. KG, Dr.-Karl-Storz-Straße 34, 78532, Tuttlingen, Germany
| | - Martin Apitz
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Annette Kopp-Schneider
- Division of Biostatistics, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 581, 69120, Heidelberg, Germany
| | - Stefanie Speidel
- Division of Translational Surgical Oncology, National Center for Tumor Diseases, Partner Site Dresden, Fetscherstraße 74, 01307, Dresden, Germany
- Centre for Tactile Internet with Human-in-the-Loop (CeTI), TU Dresden, 01307, Dresden, Germany
| | - Felix Nickel
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Pascal Probst
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Hannes G Kenngott
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany
| | - Beat P Müller-Stich
- Department for General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Im Neuenheimer Feld 420, 69120, Heidelberg, Germany.
| |
Collapse
|
9
|
Gröhl J, Kirchner T, Adler TJ, Hacker L, Holzwarth N, Hernández-Aguilera A, Herrera MA, Santos E, Bohndiek SE, Maier-Hein L. Learned spectral decoloring enables photoacoustic oximetry. Sci Rep 2021; 11:6565. [PMID: 33753769 PMCID: PMC7985523 DOI: 10.1038/s41598-021-83405-8] [Citation(s) in RCA: 26] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/19/2020] [Accepted: 01/27/2021] [Indexed: 01/15/2023] Open
Abstract
The ability of photoacoustic imaging to measure functional tissue properties, such as blood oxygenation sO[Formula: see text], enables a wide variety of possible applications. sO[Formula: see text] can be computed from the ratio of oxyhemoglobin HbO[Formula: see text] and deoxyhemoglobin Hb, which can be distuinguished by multispectral photoacoustic imaging due to their distinct wavelength-dependent absorption. However, current methods for estimating sO[Formula: see text] yield inaccurate results in realistic settings, due to the unknown and wavelength-dependent influence of the light fluence on the signal. In this work, we propose learned spectral decoloring to enable blood oxygenation measurements to be inferred from multispectral photoacoustic imaging. The method computes sO[Formula: see text] pixel-wise, directly from initial pressure spectra [Formula: see text], which represent initial pressure values at a fixed spatial location [Formula: see text] over all recorded wavelengths [Formula: see text]. The method is compared to linear unmixing approaches, as well as pO[Formula: see text] and blood gas analysis reference measurements. Experimental results suggest that the proposed method is able to obtain sO[Formula: see text] estimates from multispectral photoacoustic measurements in silico, in vitro, and in vivo.
Collapse
Affiliation(s)
- Janek Gröhl
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany.
- Medical Faculty, Heidelberg University, Heidelberg, Germany.
| | - Thomas Kirchner
- Institute of Applied Physics, Biomedical Photonics, Bern University, Bern, Switzerland
| | - Tim J Adler
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Lina Hacker
- Department of Physics, University of Cambridge, JJ Thomson Avenue, Cambridge, CB3 0HE, UK
| | - Niklas Holzwarth
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany
- Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
| | | | - Mildred A Herrera
- Department of Neurosurgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Edgar Santos
- Department of Neurosurgery, Heidelberg University Hospital, Heidelberg, Germany
| | - Sarah E Bohndiek
- Department of Physics, University of Cambridge, JJ Thomson Avenue, Cambridge, CB3 0HE, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Robinson Way, Cambridge, CB2 0RE, UK
| | - Lena Maier-Hein
- Computer Assisted Medical Interventions, German Cancer Research Center, Heidelberg, Germany.
- Medical Faculty, Heidelberg University, Heidelberg, Germany.
| |
Collapse
|
10
|
Ayala L, Seidlitz S, Vemuri A, Wirkert SJ, Kirchner T, Adler TJ, Engels C, Teber D, Maier-Hein L. Light source calibration for multispectral imaging in surgery. Int J Comput Assist Radiol Surg 2020; 15:1117-1125. [PMID: 32535848 PMCID: PMC7316688 DOI: 10.1007/s11548-020-02195-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2019] [Accepted: 04/29/2020] [Indexed: 11/15/2022]
Abstract
PURPOSE Live intra-operative functional imaging has multiple potential clinical applications, such as localization of ischemia, assessment of organ transplantation success and perfusion monitoring. Recent research has shown that live monitoring of functional tissue properties, such as tissue oxygenation and blood volume fraction, is possible using multispectral imaging in laparoscopic surgery. While the illuminant spectrum is typically kept constant in laparoscopic surgery and can thus be estimated from preoperative calibration images, a key challenge in open surgery originates from the dynamic changes of lighting conditions. METHODS The present paper addresses this challenge with a novel approach to light source calibration based on specular highlight analysis. It involves the acquisition of low-exposure time images serving as a basis for recovering the illuminant spectrum from pixels that contain a dominant specular reflectance component. RESULTS Comprehensive in silico and in vivo experiments with a range of different light sources demonstrate that our approach enables an accurate and robust recovery of the illuminant spectrum in the field of view of the camera, which results in reduced errors with respect to the estimation of functional tissue properties. Our approach further outperforms state-of-the-art methods proposed in the field of computer vision. CONCLUSION Our results suggest that low-exposure multispectral images are well suited for light source calibration via specular highlight analysis. This work thus provides an important first step toward live functional imaging in open surgery.
Collapse
Affiliation(s)
- Leonardo Ayala
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Silvia Seidlitz
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
- HIDSS4Health – Helmholtz Information and Data Science School for Health, Karlsruhe, Heidelberg, Germany
| | - Anant Vemuri
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sebastian J. Wirkert
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Thomas Kirchner
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tim J. Adler
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany
| | - Christina Engels
- Urologische Klinik, Städtisches Klinikum Karlsruhe, Karlsruhe, Germany
| | - Dogu Teber
- Urologische Klinik, Städtisches Klinikum Karlsruhe, Karlsruhe, Germany
| | - Lena Maier-Hein
- Division of Computer Assisted Medical Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Medical Faculty, Heidelberg University, Heidelberg, Germany
| |
Collapse
|
11
|
Adler TJ, Ardizzone L, Vemuri A, Ayala L, Gröhl J, Kirchner T, Wirkert S, Kruse J, Rother C, Köthe U, Maier-Hein L. Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks. Int J Comput Assist Radiol Surg 2019; 14:997-1007. [PMID: 30903566 DOI: 10.1007/s11548-019-01939-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2019] [Accepted: 03/07/2019] [Indexed: 10/27/2022]
Abstract
PURPOSE Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed. METHODS We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors. RESULTS Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera. CONCLUSION Our method could help to optimize optical camera design in an application-specific manner.
Collapse
Affiliation(s)
- Tim J Adler
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany. .,Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany.
| | | | - Anant Vemuri
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Leonardo Ayala
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Janek Gröhl
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany.,Medical Faculty, Heidelberg University, Heidelberg, Germany
| | - Thomas Kirchner
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany.,Faculty of Physics and Astronomy, Heidelberg University, Heidelberg, Germany
| | - Sebastian Wirkert
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| | - Jakob Kruse
- Visual Learning Lab, Heidelberg University, Heidelberg, Germany
| | - Carsten Rother
- Visual Learning Lab, Heidelberg University, Heidelberg, Germany
| | - Ullrich Köthe
- Visual Learning Lab, Heidelberg University, Heidelberg, Germany
| | - Lena Maier-Hein
- Computer Assisted Medical Interventions, Deutsches Krebsforschungszentrum, Im Neuenheimer Feld 223, 69120, Heidelberg, Germany
| |
Collapse
|