A Baseline Solution for the ISBI 2024 Dreaming Challenge

Diminished Reality is a technique for the removal of objects from the surroundings, providing a better view of otherwise obstructed features. This work explores the application of Diminished Reality in a medical setting, specifically aiming to visually eliminate surgical tools from operation sites for improved visibility and inspection. To this end, we introduce the Diminished Reality for Emerging Applications in Medicine through Inpainting (DREAMING) challenge. For the challenge, we created a dataset of simulated surgery scenes with obstructions from surgical instruments and hands. We evaluate an existing video inpainting model to generate diminished images from our dataset to set a baseline for the challenge. We show that the off-the-shelf model already delivers satisfactory results across some conditions, but several limitations prohibit its application for the proposed real-world scenario.