Projects | IKIM MML

3D Surgical Tool Collection

Mon, 01 Jan 0001 00:00:00 +0000

With our structured-light 3D scanners, namely the Artec Leo and AutoScan Inspec, we have scanned over 100 instruments commonly used in clinical routines. This extensive collection serves various purposes, including general instrument detection and tracking in operating room settings, as well as medical simulation or training scenarios in virtual and mixed reality. These surface models can be enhanced to achieve realism using shaders available in popular game engines like Unity3D or through 3D modeling software such as Blender.

Luijten G, Gsaxner C, Li J, Pepe A, Ambigapathy N, Kim M, et al. 3D surgical instrument collection for computer vision and extended reality. Scientific Data. 2023 Nov 11;10(1):796. Article

Watch the collection on YouTube — 3D-COSI, the 3D Collection of Surgical Instruments.

3D Visualization of CT and Segmented Models

Mon, 01 Jan 0001 00:00:00 +0000

This AR technique enables direct visualization of CT data and models within the patient, facilitating the translation from 2D CT slices to a 3D representation for enhanced diagnosis and treatment planning.

Gsaxner C, Pepe A, Li J, Ibrahimpasic U, Wallner J, Schmalstieg D, et al. Augmented Reality for Head and Neck Carcinoma Imaging: Description and Feasibility of an Instant Calibration, Markerless Approach. Computer Methods and Programs in Biomedicine 200, 105854 (2021). Article

Watch the demonstration on YouTube — CT models visualized on a 3D print with the HoloLens 2 via markerless registration.

DREAMING Challenge

Mon, 01 Jan 0001 00:00:00 +0000

We explore the potential of Diminished Reality (DR) in medicine, a modality that removes real objects from the environment by virtually replacing them with their background. DR holds huge potential in medical applications, providing surgeons with an unobstructed view of the operation site in scenarios where space constraints or disruptive instruments hinder visibility.

We hosted the DREAMING challenge — Diminished Reality for Emerging Applications in Medicine through Inpainting — as an accepted challenge at ISBI 2024. The challenge is registered at Zenodo, the dataset is published there as well, and the challenge website carries the full details.

Watch an example on YouTube — example usage of the dataset. Watch the winning solution on YouTube.

Baseline paper: van Meegdenburg T, Luijten G, Kleesiek J, Puladi B, Ferreira A, Egger J, et al. A Baseline Solution for the ISBI 2024 DREAMING Challenge. In: Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI) 2024.

Winning method: Wu ZJ, Seshimo R, Saito H, Isogawa M. Divide and Conquer: Video Inpainting for Diminished Reality in Low-Resource Settings. In: Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI) 2024.

HINTS Eye Tracking

Mon, 01 Jan 0001 00:00:00 +0000

Dizziness is a common complaint in clinical practice, with causes ranging from benign vestibular disorders to life-threatening central etiologies, particularly stroke. Rapid identification of the underlying cause is critical for timely intervention and treatment.

Using the built-in eye tracker of the HoloLens 2, we attempt to diagnose the cause using traditional and machine learning methods. The project can provide an objective and excellent diagnosis, and thus improve or even replace the current HINTS exam. Future work focuses on bringing the method out of the hospital and into the pocket of our patients, so that quick diagnosis is available anytime, anywhere.

Watch the project video on YouTube — a German-language film produced by the Centre for Virtual and Extended Reality in Medicine (ZvRM).

Instrument Tracking

Mon, 01 Jan 0001 00:00:00 +0000

Using the sensors of the HoloLens 2, we have achieved highly accurate instrument tracking through infrared reflective markers. By combining this technology with our previous advancements, we offer an out-of-the-box solution for surgical navigation, eliminating the need for additional expensive equipment. An interesting follow-up would be the less accurate but no less interesting free-form segmentation and tracking, that is, tracking without infrared reflective markers.

Augmented reality instrument tracking with the HoloLens 2

Gsaxner C, Li J, Pepe A, Schmalstieg D, Egger J. Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality. In: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology. Osaka, Japan: ACM; 2021. p. 1–11. Article

Markerless Patient Registration

Mon, 01 Jan 0001 00:00:00 +0000

We have developed an automated markerless registration method for seamlessly integrating CT data and segmented models into the patient. This method addresses the switching-focus problem, providing surgeons with a direct 2D-to-3D mapping of CT data and enabling surgical navigation in a cost-effective manner.

Gsaxner C, Pepe A, Wallner J, Schmalstieg D, Egger J. Markerless Image-to-Face Registration for Untethered Augmented Reality in Head and Neck Surgery. In: Medical Image Computing and Computer Assisted Intervention — MICCAI 2019. p. 236–44. Article

Watch the demonstration on YouTube — markerless patient registration using the depth sensors of the HoloLens 2.

Multi-User Learning Experiences

Mon, 01 Jan 0001 00:00:00 +0000

Collaborative learning takes center stage in our research, as we create immersive multi-user experiences that foster interactive education and training for medical students. It can also be leveraged to enable multidisciplinary teams to discuss a patient’s treatment plan without the need to be in the same room.

Watch the demonstration on YouTube — MultiAR, a cross-device, multi-user augmented reality platform for biomedical education.