In this presentation, I will present Metrics Reloaded - a comprehensive framework guiding researchers in the problem-aware selection of metrics in biomedical image analysis. I will first demonstrate how the choice of performance metrics often fails to align with the clinicians’ interests, thereby impeding scientific progress and the practical application of machine learning (ML) algorithms. Next, I will present the Metrics Reloaded framework, which has been developed by an international expert consortium to overcome the aforementioned issues.
Intelligent medical systems adept at acquiring and analyzing sensor data to offer context-sensitive support are at the forefront of modern healthcare. However, various factors, often not immediately apparent, significantly hinder the effective integration of contemporary machine learning research into clinical practice. Using insights from my own research team and extensive international collaborations, I will delve into prevalent issues in current medical imaging practices and offer potential remedies. My talk will highlight the vital importance of challenging every aspect of the medical imaging pipeline from the image modalities applied to the validation methodology, ensuring that intelligent imaging systems are primed for genuine clinical implementation.
The generation of realistically simulated photoacoustic (PA) images with ground truth labels for optical and acoustic properties has become a critical method for training and validating neural networks for PA imaging. As state-of-the-art model-based simulations often suffer from various inaccuracies, unsupervised domain transfer methods have been recently proposed to enhance the quality of model-based simulations. The validation of these methods, however, is challenging as there are no reliable labels for absorption or oxygen saturation in vivo. In this work, we examine various domain shifts between simulations and real images such as simulating the wrong noise model, inaccuracies in modeling the digital device twin or erroneous assumptions on tissue composition. We show in silico how a Cycle GAN, unsupervised image-to-image translation networks (UNIT) and a conditional invertible neural network handle these domain shifts and what their consequences are for blood oxygen saturation estimation.
This study delves into the largely uncharted domain of biases in photoacoustic imaging, spotlighting potential shortcut learning as a key issue in reliable machine learning. Our focus is on hardware variation biases. We identify device-specific traits that create detectable fingerprints in photoacoustic images, demonstrate machine learning's capability to use these discrepancies to determine the device that acquired the image, and highlight their potential impact on machine learning model predictions in downstream tasks, such as disease classification.
Data-driven approaches to the quantification problem in photoacoustic imaging have shown great potential in silico, but the inherent lack of labelled ground truth data in vivo currently restricts their application and translation into clinics. In this study we leverage Fourier Neural Operator networks as surrogate models to synthesize multispectral photoacoustic human forearm images in order to replace time-consuming and not inherently differentiable state-of-the-art Monte Carlo and k-Wave simulations. We investigate the accuracy and efficiency of these surrogate models for the optical and acoustic simulation step.
Optical and acoustic imaging techniques enable noninvasive visualization of structural and functional tissue properties. Data-driven approaches for quantification of these properties are promising, but they rely on highly accurate simulations due to the lack of ground truth knowledge. We recently introduced the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit that has quickly been adopted by the community in the context of the IPASC consortium for standardized reconstruction. We present new developments in the toolkit including e.g. improved tissue and device modeling and provide an outlook on future directions aiming at improving the realism of simulations.
Peripheral artery disease (PAD) is widespread among the elderly population where narrowing arteries in lower limbs are causing a lack of perfusion. This work explores the benefit of volumetric photoacoustic imaging (v-PAI) over conventional 2D PAI for PAD diagnosis and monitoring. To this end, we leverage the recently proposed approach of Tattoo tomography, which generates a v-PAI representation from a set of 2D PAI slices. Preliminary results of the ongoing study indicate that v-PAI can increase the sensitivity of early-stage PAD detection. Conclusively our Tattoo approach has the potential to become a valuable tool in PAD diagnostics.
KEYWORDS: Monte Carlo methods, Diffuse reflectance spectroscopy, In vivo imaging, Tissues, Optical imaging, Machine learning, Hyperspectral imaging, Functional imaging, Evolutionary algorithms, Data analysis
Simulations are indispensable in the field of biomedical optical imaging, particularly in functional imaging. Given the recent rise of artificial intelligence and the lack of labeled in vivo data, synthetic data is not only important for the validation of algorithms but also crucial for training machine learning methods. To support research based on synthetic data, we present a new framework for assessing the quality of synthetic spectral data. Experiments with more than 10,000 hyperspectral in vivo images obtained from multiple species and various organ classes indicate that our framework could become an important tool for researchers working with simulations.
Significance: Optical and acoustic imaging techniques enable noninvasive visualisation of structural and functional properties of tissue. The quantification of measurements, however, remains challenging due to the inverse problems that must be solved. Emerging data-driven approaches are promising, but they rely heavily on the presence of high-quality simulations across a range of wavelengths due to the lack of ground truth knowledge of tissue acoustical and optical properties in realistic settings.
Aim: To facilitate this process, we present the open-source simulation and image processing for photonics and acoustics (SIMPA) Python toolkit. SIMPA is being developed according to modern software design standards.
Approach: SIMPA enables the use of computational forward models, data processing algorithms, and digital device twins to simulate realistic images within a single pipeline. SIMPA’s module implementations can be seamlessly exchanged as SIMPA abstracts from the concrete implementation of each forward model and builds the simulation pipeline in a modular fashion. Furthermore, SIMPA provides comprehensive libraries of biological structures, such as vessels, as well as optical and acoustic properties and other functionalities for the generation of realistic tissue models.
Results: To showcase the capabilities of SIMPA, we show examples in the context of photoacoustic imaging: the diversity of creatable tissue models, the customisability of a simulation pipeline, and the degree of realism of the simulations.
Conclusions: SIMPA is an open-source toolkit that can be used to simulate optical and acoustic imaging modalities. The code is available at: https://github.com/IMSY-DKFZ/simpa, and all of the examples and experiments in this paper can be reproduced using the code available at: https://github.com/IMSY-DKFZ/simpa_paper_experiments.
KEYWORDS: Data modeling, Monte Carlo methods, Scattering, Photoacoustic imaging, Optical simulations, Neural networks, Machine learning, Light scattering, In vivo imaging, Imaging spectroscopy
Photoacoustic imaging (PAI) has the potential to revolutionize healthcare due to the valuable information on tissue physiology that is contained in multispectral signals. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. In this work, we present a deep learning-based approach to semantic segmentation of multispectral PA images to facilitate interpretability of recorded images. Based on a validation study with experimentally acquired data of healthy human volunteers, we show that a combination of tissue segmentation, sO2 estimation, and uncertainty quantification can create powerful analyses and visualizations of multispectral photoacoustic images.
In this work, we present the open source “Simulation and Image Processing for Photoacoustic Imaging (SIMPA)” toolkit that facilitates simulation of multispectral photoacoustic images by streamlining the use of state-of-the-art frameworks that numerically approximate the respective forward models. SIMPA provides modules for all the relevant steps for photoacoustic forward simulation: tissue modelling, optical forward modelling, acoustic modelling, noise modelling, as well as image reconstruction. We demonstrate the capabilities of SIMPA by performing image simulation using MCX and k-Wave for the optical and acoustic forward modelling, as well as an experimentally determined noise model and a custom tissue model.
Previous work on 3D freehand photoacoustic imaging has focused on the development of specialized hardware or the use of tracking devices. In this work, we present a novel approach towards 3D volume compounding using an optical pattern attached to the skin. By design, the pattern allows context-aware calculation of the PA image pose in a pattern reference frame, enabling 3D reconstruction while also making the method robust against patient motion. Due to its easy handling optical pattern-enabled context-aware PA imaging could be a promising approach for 3D PA in a clinical environment.
Photoacoustic imaging (PAI) is an emerging medical imaging modality that provides high contrast and spatial resolution. A core unsolved problem to effectively support interventional healthcare is the accurate quantification of the optical tissue properties, such as the absorption and scattering coefficients. The contribution of this work is two-fold. We demonstrate the strong dependence of deep learning-based approaches on the chosen training data and we present a novel approach to generating simulated training data. According to initial in silico results, our method could serve as an important first step related to generating adequate training data for PAI applications.
Photoacoustics Imaging is an emerging imaging modality enabling the recovery of functional tissue parameters such as blood oxygenation. However, quantifying these still remains challenging mainly due to the non-linear influence of the light fluence which makes the underlying inverse problem ill-posed. We tackle this gap with invertible neural networks and present a novel approach to quantifying uncertainties related to reconstructing physiological parameters, such as oxygenation. According to in silico experiments, blood oxygenation prediction with invertible neural networks combined with an interactive visualization could serve as a powerful method to investigate the effect of spectral coloring on blood oxygenation prediction tasks.
One of the major applications of multispectral photoacoustic imaging is the recovery of functional tissue properties with the goal of distinguishing different tissue classes. In this work, we tackle this challenge by employing a deep learning-based algorithm called learned spectral decoloring for quantitative photoacoustic imaging. With the combination of tissue classification, sO2 estimation, and uncertainty quantification, powerful analyses and visualizations of multispectral photoacoustic images can be created. Consequently, these could be valuable tools for the clinical translation of photoacoustic imaging.
Optical imaging for estimating physiological parameters, such as tissue oxygenation or blood volume fraction has been an active field of research for many years. In this context, machine learning -based approaches are gaining increasing attention in the literature. Following up on this trend, this talk will present recent progress in multispectral optical and photoacoustic image analysis using deep learning (DL). From a methodological point of view, it will focus on two challenges: (1) How to train a DL algorithm in the absence of reliable reference training data and (2) how to quantify and compensate the different types of uncertainties associated with the inference of physiological parameters. The research presented is being conducted in the scope of the European Research Council (ERC) starting grant COMBIOSCOPY.
The International Photoacoustic Standardisation Consortium (IPASC) emerged from SPIE 2018, established to drive consensus on photoacoustic system testing. As photoacoustic imaging (PAI) matures from research laboratories into clinical trials, it is essential to establish best-practice guidelines for photoacoustic image acquisition, analysis and reporting, and a standardised approach for technical system validation. The primary goal of the IPASC is to create widely accepted phantoms for testing preclinical and clinical PAI systems. To achieve this, the IPASC has formed five working groups (WGs). The first and second WGs have defined optical and acoustic properties, suitable materials, and configurations of photoacoustic image quality phantoms. These phantoms consist of a bulk material embedded with targets to enable quantitative assessment of image quality characteristics including resolution and sensitivity across depth. The third WG has recorded details such as illumination and detection configurations of PAI instruments available within the consortium, leading to proposals for system-specific phantom geometries. This PAI system inventory was also used by WG4 in identifying approaches to data collection and sharing. Finally, WG5 investigated means for phantom fabrication, material characterisation and PAI of phantoms. Following a pilot multi-centre phantom imaging study within the consortium, the IPASC settled on an internationally agreed set of standardised recommendations and imaging procedures. This leads to advances in: (1) quantitative comparison of PAI data acquired with different data acquisition and analysis methods; (2) provision of a publicly available reference data set for testing new algorithms; and (3) technical validation of new and existing PAI devices across multiple centres.
Multispectral photoacoustic (PA) imaging is a prime modality to monitor hemodynamics and changes in blood oxygenation (sO2). Although sO2 changes can be an indicator of brain activity both in normal and in pathological conditions, PA imaging of the brain has mainly focused on small animal models with lissencephalic brains. Therefore, the purpose of this work was to investigate the usefulness of multispectral PA imaging in assessing sO2 in a gyrencephalic brain. To this end, we continuously imaged a porcine brain as part of an open neurosurgical intervention with a handheld PA and ultrasonic (US) imaging system in vivo. Throughout the experiment, we varied respiratory oxygen and continuously measured arterial blood gases. The arterial blood oxygenation (SaO2) values derived by the blood gas analyzer were used as a reference to compare the performance of linear spectral unmixing algorithms in this scenario. According to our experiment, PA imaging can be used to monitor sO2 in the porcine cerebral cortex. While linear spectral unmixing algorithms are well-suited for detecting changes in oxygenation, there are limits with respect to the accurate quantification of sO2, especially in depth. Overall, we conclude that multispectral PA imaging can potentially be a valuable tool for change detection of sO2 in the cerebral cortex of a gyrencephalic brain. The spectral unmixing algorithms investigated in this work will be made publicly available as part of the open-source software platform Medical Imaging Interaction Toolkit (MITK).
Eric Heim, Tobias Roß, Alexander Seitel, Keno März, Bram Stieltjes, Matthias Eisenmann, Johannes Lebert, Jasmin Metzger, Gregor Sommer, Alexander W. Sauter, Fides Regina Schwartz, Andreas Termer, Felix Wagner, Hannes Götz Kenngott, Lena Maier-Hein
Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.
Real-time monitoring of functional tissue parameters, such as local blood oxygenation, based on optical imaging could provide groundbreaking advances in the diagnosis and interventional therapy of various diseases. Although photoacoustic (PA) imaging is a modality with great potential to measure optical absorption deep inside tissue, quantification of the measurements remains a major challenge. We introduce the first machine learning-based approach to quantitative PA imaging (qPAI), which relies on learning the fluence in a voxel to deduce the corresponding optical absorption. The method encodes relevant information of the measured signal and the characteristics of the imaging system in voxel-based feature vectors, which allow the generation of thousands of training samples from a single simulated PA image. Comprehensive in silico experiments suggest that context encoding-qPAI enables highly accurate and robust quantification of the local fluence and thereby the optical absorption from PA images.
KEYWORDS: Reconstruction algorithms, Photoacoustic spectroscopy, Signal to noise ratio, Ultrasonography, Transducers, Image resolution, Pulsed laser operation, Chromophores, In vitro testing, In vivo imaging
Reconstruction of photoacoustic images acquired with clinical ultrasound transducers is traditionally performed using the delay and sum (DAS) beamforming algorithm. Recently, the delay multiply and sum (DMAS) beamforming algorithm has been shown to provide increased contrast, signal to noise ratio (SNR) and resolution in PA imaging. The main reason for the continued use of DAS beamforming in photoacoustics is its linearity in reconstructing the PA signal to the initial pressure generated by the absorbed laser pulse. This is crucial for the identification of different chromophores in multispectral PA applications and DMAS has not yet been demonstrated to provide this property. Furthermore, due to its increased computational complexity, DMAS has not yet been shown to work in real time.
We present an open-source real-time variant of the DMAS algorithm which ensures linearity of the reconstruction while still providing increased SNR and therefore enables use of DMAS for multispectral PA applications. This is demonstrated in vitro and in vivo. The DMAS and reference DAS algorithms were integrated in the open-source Medical Imaging Interaction Toolkit (MITK) and are available to the community as real-time capable GPU implementations.
KEYWORDS: Software development, Blood, Photoacoustic spectroscopy, In vivo imaging, Ultrasonography, Imaging systems, Scanners, Medical imaging, Control systems, Ultrasonics
Photoacoustic (PA) systems based on clinical linear ultrasound arrays have become increasingly popular in translational PA research. Such systems can more easily be integrated in a clinical workflow due to the simultaneous access to ultrasonic imaging and their familiarity of use to clinicians. In contrast to more complex setups, handheld linear probes can be applied to a large variety of clinical use cases. However, most translational work with such scanners is based on proprietary development and as such not accessible to the community.
In this contribution, we present a custom-built, hybrid, multispectral, real-time photoacoustic and ultrasonic imaging system with a linear array probe that is controlled by software developed within the highly customisable and extendable open-source software platform Medical Imaging Interaction Toolkit (MITK). Our software offers direct control of both the laser and the ultrasonic system and may thus serve as a starting point for various translational research and development. To demonstrate the extensibility of our system, we developed an open-source software plugin for real-time in vivo blood oxygenation measurements. Blood oxygenation is estimated by spectral unmixing of hemoglobin chromophores. The performance is demonstrated on in vivo measurements of the common carotid artery as well as peripheral extremity vessels of healthy volunteers.
KEYWORDS: Multispectral imaging, Monte Carlo methods, Tissue optics, Surgery, Tumors, Blood, In vivo imaging, Laparoscopy, Machine learning, Image resolution
Multispectral imaging (MSI) could be useful for many applications in surgery, including tumor detection and perfusion monitoring. Acquisition of many bands however leads to long imaging times and/or low resolution, hampering widespread adoption of the technique. To overcome this issue, current research focusses on reducing the number of recorded bands. Yet, the methods proposed are not able to consider both the target domain (e.g. liver surgery) and the specific task (e.g. oxygenation or blood volume fraction monitoring) when selecting bands.
In this work we present the first approach to domain and task specific band selection. Our method relies on highly generic Monte Carlo-based tissue simulations that aim to capture a large range of optical tissue parameters potentially observed during surgical interventions. The adaptation of the model to a specific clinical application is based on label-free in vivo hyperspectral recordings using a recently published approach to multispectral domain adaptation. The bands are selected based on their performance to estimate a task-dependent physiological parameter. This performance is evaluated on the adapted simulations, which come with ground truth values. According to in vivo experiments with hyperspectral recordings of tumors in a mouse model, a small subset of bands is enough for accurate oxygenation and blood volume fraction estimation. Compared to state-of-the-art baseline methods, bands selected by our method show more accurate results in oxygenation estimation. Our work could thus help remove one of the last barriers for interventional usage of MSI.
KEYWORDS: Sensors, Monte Carlo methods, Image processing, Photoacoustic spectroscopy, Reconstruction algorithms, Computer simulations, Error analysis, Data modeling, Tissues, Medical imaging
Quantification of tissue properties with photoacoustic (PA) imaging typically requires a highly accurate representation of the initial pressure distribution in tissue. Almost all PA scanners reconstruct the PA image only from a partial scan of the emitted sound waves. Especially handheld devices, which have become increasingly popular due to their versatility and ease of use, only provide limited view data because of their geometry. Owing to such limitations in hardware as well as to the acoustic attenuation in tissue, state-of-the-art reconstruction methods deliver only approximations of the initial pressure distribution. To overcome the limited view problem, we present a machine learning-based approach to the reconstruction of initial pressure from limited view PA data. Our method involves a fully convolutional deep neural network based on a U-Net-like architecture with pixel-wise regression loss on the acquired PA images. It is trained and validated on in silico data generated with Monte Carlo simulations. In an initial study we found an increase in accuracy over the state-of-the-art when reconstructing simulated linear-array scans of blood vessels.
Quantification of photoacoustic (PA) images is one of the major challenges currently being addressed in PA research. Tissue properties can be quantified by correcting the recorded PA signal with an estimation of the corresponding fluence. Fluence estimation itself, however, is an ill-posed inverse problem which usually needs simplifying assumptions to be solved with state-of-the-art methods. These simplifications, as well as noise and artifacts in PA images reduce the accuracy of quantitative PA imaging (PAI). This reduction in accuracy is often localized to image regions where the assumptions do not hold true. This impedes the reconstruction of functional parameters when averaging over entire regions of interest (ROI). Averaging over a subset of voxels with a high accuracy would lead to an improved estimation of such parameters. To achieve this, we propose a novel approach to the local estimation of confidence in quantitative reconstructions of PA images. It makes use of conditional probability densities to estimate confidence intervals alongside the actual quantification. It encapsulates an estimation of the errors introduced by fluence estimation as well as signal noise. We validate the approach using Monte Carlo generated data in combination with a recently introduced machine learning-based approach to quantitative PAI. Our experiments show at least a two-fold improvement in quantification accuracy when evaluating on voxels with high confidence instead of thresholding signal intensity.
Intraoperative tissue classification is one of the prerequisites for providing context-aware visualization in computer-assisted minimally invasive surgeries. As many anatomical structures are difficult to differentiate in conventional RGB medical images, we propose a classification method based on multispectral image patches. In a comprehensive ex vivo study through statistical analysis, we show that (1) multispectral imaging data are superior to RGB data for organ tissue classification when used in conjunction with widely applied feature descriptors and (2) combining the tissue texture with the reflectance spectrum improves the classification performance. The classifier reaches an accuracy of 98.4% on our dataset. Multispectral tissue analysis could thus evolve as a key enabling technique in computer-assisted laparoscopy.
Andreas Fetzer, Jasmin Metzger, Darko Katic, Keno März, Martin Wagner, Patrick Philipp, Sandy Engelhardt, Tobias Weller, Sascha Zelzer, Alfred Franz, Nicolai Schoch, Vincent Heuveline, Maria Maleshkova, Achim Rettinger, Stefanie Speidel, Ivo Wolf, Hannes Kenngott, Arianeb Mehrabi, Beat Müller-Stich, Lena Maier-Hein, Hans-Peter Meinzer, Marco Nolden
KEYWORDS: Surgery, Data integration, Data modeling, Imaging informatics, Data storage, Standards development, Data modeling, Knowledge acquisition, Cognition, Knowledge management, Medical imaging, Image segmentation, Picture Archiving and Communication System, Information science, Neuroimaging, Data archive systems
In the surgical domain, individual clinical experience, which is derived in large part from past clinical cases, plays
an important role in the treatment decision process. Simultaneously the surgeon has to keep track of a large
amount of clinical data, emerging from a number of heterogeneous systems during all phases of surgical treatment.
This is complemented with the constantly growing knowledge derived from clinical studies and literature. To
recall this vast amount of information at the right moment poses a growing challenge that should be supported
by adequate technology.
While many tools and projects aim at sharing or integrating data from various sources or even provide knowledge-based
decision support - to our knowledge - no concept has been proposed that addresses the entire surgical
pathway by accessing the entire information in order to provide context-aware cognitive assistance. Therefore a
semantic representation and central storage of data and knowledge is a fundamental requirement.
We present a semantic data infrastructure for integrating heterogeneous surgical data sources based on a common
knowledge representation. A combination of the Extensible Neuroimaging Archive Toolkit (XNAT) with semantic
web technologies, standardized interfaces and a common application platform enables applications to access and
semantically annotate data, perform semantic reasoning and eventually create individual context-aware surgical
assistance.
The infrastructure meets the requirements of a cognitive surgical assistant system and has been successfully
applied in various use cases. The system is based completely on free technologies and is available to the community
as an open-source package.
Michael Goetz, Eric Heim, Keno Maerz, Tobias Norajitra, Mohammadreza Hafezi, Nassim Fard, Arianeb Mehrabi, Max Knoll, Christian Weber, Lena Maier-Hein, Klaus Maier-Hein
Current fully automatic liver tumor segmentation systems are designed to work on a single CT-image. This hinders these systems from the detection of more complex types of liver tumor. We therefore present a new algorithm for liver tumor segmentation that allows incorporating different CT scans and requires no manual interaction. We derive a liver segmentation with state-of-the-art shape models which are robust to initialization. The tumor segmentation is then achieved by classifying all voxels into healthy or tumorous tissue using Extremely Randomized Trees with an auto-context learning scheme. Using DALSA enables us to learn from only sparse annotations and allows a fast set-up for new image settings. We validate the quality of our algorithm with exemplary segmentation results.
Intra-operative tissue classification is one of the prerequisites for providing context-aware visualization in computer-assisted minimally invasive surgeries. As many anatomical structures are difficult to differentiate in conventional RGB medical images, we propose a classification method based on multispectral image patches. In a comprehensive ex vivo study we show (1) that multispectral imaging data is superior to RGB data for organ tissue classification when used in conjunction with widely applied feature descriptors and (2) that combining the tissue texture with the reflectance spectrum improves the classification performance. Multispectral tissue analysis could thus evolve as a key enabling technique in computer-assisted laparoscopy.
3D Visualization of anatomical data is an integral part of diagnostics and treatment in many medical disciplines, such as radiology, surgery and forensic medicine. To enable intuitive interaction with the data, we recently proposed a new concept for on-patient visualization of medical data which involves rendering of subsurface structures on a mobile display that can be moved along the human body. The data fusion is achieved with a range imaging device attached to the display. The range data is used to register static 3D medical imaging data with the patient body based on a surface matching algorithm. However, our previous prototype was based on the Microsoft Kinect camera and thus required a cable connection to acquire color and depth data. The contribution of this paper is two-fold. Firstly, we replace the Kinect with the Structure Sensor - a novel cable-free range imaging device - to improve handling and user experience and show that the resulting accuracy (target registration error: 4.8±1.5 mm) is comparable to that achieved with the Kinect. Secondly, a new approach to visualizing complex 3D anatomy based on this device, as well as 3D printed models of anatomical surfaces, is presented. We demonstrate that our concept can be applied to in vivo data and to a 3D printed skull of a forensic case. Our new device is the next step towards clinical integration and shows that the concept cannot only be applied during autopsy but also for presentation of forensic data to laypeople in court or medical education.
KEYWORDS: Photoacoustic tomography, Angiography, Acquisition tracking and pointing, Tissues, Signal processing, Calibration, Tissue optics, 3D image processing, Optical imaging, Signal to noise ratio, 3D image reconstruction, Absorption, Imaging systems, Visualization
Photo-acoustic tomography (PAT) is capable of imaging optical absorption in depths beyond the diffusion limit. As blood is one of the main absorbers in tissue, one important application is the visualization of vasculature, which can provide important clues for diagnosing diseases like cancer. While the state-of-the-art work in photo-acoustic 3D angiography has focused on computed tomography systems involving complex setups, we propose an approach based on optically tracking a freehand linear ultrasound probe that can be smoothly integrated into the clinical workflow. To this end, we present a method for calibration of a PAT system using an N-wire phantom specifically designed for PAT and show how to use local gradient information in the 3D reconstructed volume to significantly enhance the signal. According to experiments performed with a tissue mimicking intra-lipid phantom, the signal-to-noise ratio, contrast and contrast-to-noise ratio measured in the full field of view of the linear probe can be improved by factors of 1.7±0.7, 14.6±5.8 and 2.8±1.2 respectively, when comparing the post envelope detection reconstructed 3D volume with the processed one. Qualitative validation performed in tissue mimicking gelatin phantoms further showed good agreement of the reconstructed vasculature with corresponding structures extracted from X-ray computed tomographies. As our method provides high contrast 3D images of the vasculature despite a low hardware complexity its potential for clinical application is high.
Radiotherapy is frequently used to treat unoperated or partially resected tumors. Tumor movement, e.g. caused by respiration, is a major challenge in this context. Markers can be implanted around the tumor prior to radiation therapy for accurate tracking of tumor movement. However, accurate placement of these markers while keeping a secure margin around the target and while taking into account critical structures is a difficult task. Computer-assisted needle insertion has been an active field of research in the past decades. However, the challenge of navigated marker placement for motion compensated radiotherapy has not yet been addressed. This work presents a system to support marker implantation for radiotherapy under consideration of safety margins and optimal marker configuration. It is designed to allow placement of markers both percutaneously and during an open liver surgery. To this end, we adapted the previously proposed EchoTrack system which integrates ultrasound (US) imaging and electromagnetic (EM) tracking in a single mobile modality. The potential of our new marker insertion concept was evaluated in a phantom study by inserting sets of three markers around dedicated targets (n=22) simultaneously spacing the markers evenly around the target as well as placing the markers in a defined distance to the target. In all cases the markers were successfully placed in a configuration fulfilling the predefined criteria. This includes a minimum distance of 18.9 ± 2.4 mm between marker and tumor as well as a divergence of 2.1 ± 1.5 mm from the planned marker positions. We conclude that our system has high potential to facilitate the placement of markers in suitable configurations for surgeons without extensive experience in needle punctions as high quality configurations were obtained even by medical non-experts.
A. Franz, A. Seitel, M. Servatius, C. Zöllner, I. Gergel, I. Wegner, J. Neuhaus, S. Zelzer, M. Nolden, J. Gaa, P. Mercea, K. Yung, C. Sommer, B. Radeleff, H.-P. Schlemmer, H.-U. Kauczor, H.-P. Meinzer, L. Maier-Hein
Due to rapid developments in the research areas of medical imaging, medical image processing and robotics,
computer assistance is no longer restricted to diagnostics and surgical planning but has been expanded to surgical
and radiological interventions. From a software engineering point of view, the systems for image-guided therapy
(IGT) are highly complex. To address this issue, we presented an open source extension to the well-known
Medical Imaging Interaction Toolkit (MITK) for developing IGT systems, called MITK-IGT. The contribution
of this paper is two-fold: Firstly, we extended MITK-IGT such that it (1) facilitates the handling of navigation
tools, (2) provides reusable graphical user interface (UI) components, and (3) features standardized exception
handling. Secondly, we developed a software prototype for computer-assisted needle insertions, using the new
features, and tested it with a new Tabletop field generator (FG) for the electromagnetic tracking system NDI
Aurora ®. To our knowledge, we are the first to have integrated this new FG into a complete navigation system
and have conducted tests under clinical conditions. In conclusion, we enabled simplified development of imageguided
therapy software and demonstrated the utilizability of applications developed with MITK-IGT in the
clinical workflow.
Augmented Reality (AR) is a convenient way of porting information from medical images into the surgical field of
view and can deliver valuable assistance to the surgeon, especially in laparoscopic procedures. In addition, high
definition (HD) laparoscopic video devices are a great improvement over the previously used low resolution
equipment. However, in AR applications that rely on real-time detection of fiducials from video streams, the demand
for efficient image processing has increased due to the introduction of HD devices. We present an algorithm based on
the well-known Conditional Density Propagation (CONDENSATION) algorithm which can satisfy these new
demands. By incorporating a prediction around an already existing and robust segmentation algorithm, we can speed
up the whole procedure while leaving the robustness of the fiducial segmentation untouched. For evaluation purposes
we tested the algorithm on recordings from real interventions, allowing for a meaningful interpretation of the results.
Our results show that we can accelerate the segmentation by a factor of 3.5 on average. Moreover, the prediction
information can be used to compensate for fiducials that are temporarily occluded or out of scope, providing greater
stability.
Visualization of anatomical data for disease diagnosis, surgical planning, or orientation during interventional
therapy is an integral part of modern health care. However, as anatomical information is typically shown on
monitors provided by a radiological work station, the physician has to mentally transfer internal structures
shown on the screen to the patient. To address this issue, we recently presented a new approach to on-patient
visualization of 3D medical images, which combines the concept of augmented reality (AR) with an intuitive
interaction scheme. Our method requires mounting a range imaging device, such as a Time-of-Flight (ToF)
camera, to a portable display (e.g. a tablet PC). During the visualization process, the pose of the camera and
thus the viewing direction of the user is continuously determined with a surface matching algorithm. By moving
the device along the body of the patient, the physician is given the impression of looking directly into the human
body. In this paper, we present and evaluate a new method for camera pose estimation based on an anisotropic
trimmed variant of the well-known iterative closest point (ICP) algorithm. According to in-silico and in-vivo
experiments performed with computed tomography (CT) and ToF data of human faces, knees and abdomens,
our new method is better suited for surface registration with ToF data than the established trimmed variant of
the ICP, reducing the target registration error (TRE) by more than 60%. The TRE obtained (approx. 4-5 mm)
is promising for AR visualization, but clinical applications require maximization of robustness and run-time.
KEYWORDS: Computed tomography, Error analysis, Time of flight cameras, Data modeling, Data acquisition, Image registration, Natural surfaces, Target detection, Magnetic resonance imaging, Reliability
Range imaging modalities, such as time-of-flight cameras (ToF), are becoming very popular for the acquisition of intra-operative
data, which can be used for registering the patient's anatomy with pre-operative data, such as 3D images generated
by computed tomographies (CT) or magnetic resonance imaging (MRI). However, due to the distortions that appear because
of the different acquisition principles of the input surfaces, the noise, and the deformations that may occur in the intra-operative
environment, we face different surface properties for points lying on the same anatomical locations and unreliable
feature points detection, which are crucial for most surface matching algorithms. In order to overcome these issues,
we present a method for automatically finding correspondences between surfaces that searches for minimally deformed
configurations. For this purpose, an error metric that expresses the reliability of a correspondence set based on its spatial
configuration is employed. The registration error is minimized by a combinatorial analysis through search-trees. Our
method was evaluated with real and simulated ToF and CT data, and showed to be reliable for the registration of partial
multi-modal surfaces with noise and distortions.
KEYWORDS: Medical imaging, Algorithm development, Image processing, Current controlled current source, Cameras, 3D acquisition, 3D image processing, 3D-TOF imaging, Image registration, In vitro testing
Time-of-flight (ToF) cameras are a novel, fast, and robust means for intra-operative 3D surface acquisition. They acquire surface information (range images) in real-time. In the intra-operative registration context, these surfaces must be matched to pre-operative CT or MR surfaces, using so called descriptors, which represent surface characteristics. We present a framework for local and global multi-modal comparison of surface descriptors and characterize the differences between ToF and CT data in an in vitro experiment. The framework takes into account various aspects related to the surface characteristics and does not require high resolution input data in order to establish appropriate correspondences. We show that the presentation of local and global comparison data allows for an accurate assessment of ToF-CT discrepancies. The information gained from our study may be used for developing ToF pre-processing and matching algorithms, or for improving calibration procedures for compensating systematic distance errors. The framework is available in the open-source platform Medical Imaging Interaction Toolkit (MITK).
The Iterative Closest Point (ICP) algorithm is a widely used method for geometric alignment of 3D models.
Given two roughly aligned shapes represented by two point sets, the algorithm iteratively establishes point
correspondences given the current alignment of the data and computes a rigid transformation accordingly. It
can be shown that the method converges to an at least local minimimum with respect to a mean-square distance
metric. From a statistical point of view, the algorithm implicitly assumes that the points are observed with
isotropic Gaussian noise. In this paper, we (1) present the first variant of the ICP that accounts for anisotropic
localization uncertainty in both shapes as well as in both steps of the algorithm and (2) show how to apply the
method for robust fine registration of surface meshes. According to an evaluation on medical imaging data,
the proposed method is better suited for fine surface registration than the original ICP, reducing the target
registration error (TRE) for a set of targets located inside or near the mesh by 80% on average.
Image-guided therapy systems generally require registration of pre-operative planning data with the patient's anatomy. One common approach to achieve this is to acquire intra-operative surface data and match it to surfaces extracted from the planning image. Although increasingly popular for surface generation in general, the novel Time-of-Flight (ToF) technology has not yet been applied in this context. This may be attributed to the fact that the ToF range images are subject to considerable noise. The contribution of this study is two-fold. Firstly, we present an adaption of the well-known bilateral filter for denoising ToF range images based on the noise characteristics of the camera. Secondly, we assess the quality of organ surfaces generated from ToF range data with and without bilateral smoothing using corresponding high resolution CT data as ground truth. According to an evaluation on five porcine organs, the root mean squared (RMS) distance between the denoised ToF data points and the reference computed tomography (CT) surfaces ranged from 3.0 mm (lung) to 9.0 mm (kidney). This corresponds to an error-reduction of up to 36% compared to the error of the original ToF surfaces.
One of the main challenges related to computer-assisted laparoscopic surgery is the accurate registration of
pre-operative planning images with patient's anatomy. One popular approach for achieving this involves intraoperative
3D reconstruction of the target organ's surface with methods based on multiple view geometry. The
latter, however, require robust and fast algorithms for establishing correspondences between multiple images of
the same scene. Recently, the first endoscope based on Time-of-Flight (ToF) camera technique was introduced.
It generates dense range images with high update rates by continuously measuring the run-time of intensity
modulated light. While this approach yielded promising results in initial experiments, the endoscopic ToF
camera has not yet been evaluated in the context of related work. The aim of this paper was therefore to
compare its performance with different state-of-the-art surface reconstruction methods on identical objects. For
this purpose, surface data from a set of porcine organs as well as organ phantoms was acquired with four
different cameras: a novel Time-of-Flight (ToF) endoscope, a standard ToF camera, a stereoscope, and a High
Definition Television (HDTV) endoscope. The resulting reconstructed partial organ surfaces were then compared
to corresponding ground truth shapes extracted from computed tomography (CT) data using a set of local and
global distance metrics. The evaluation suggests that the ToF technique has high potential as means for intraoperative
endoscopic surface registration.
Augmented reality (AR) for enhancement of intra-operative images is gaining increasing interest in the field of
navigated medical interventions. In this context, various imaging modalities such as ultrasound (US), C-Arm
computed tomography (CT) and endoscopic images have been applied to acquire intra-operative information
about the patient's anatomy. The aim of this paper was to evaluate the potential of the novel Time-of-Flight
(ToF) camera technique as means for markerless intra-operative registration. For this purpose, ToF range data
and corresponding CT images were acquired from a set of explanted non-transplantable human and porcine
organs equipped with a set of marker that served as targets. Based on a rigid matching of the surfaces generated
from the ToF images with the organ surfaces generated from the CT data, the targets extracted from the
planning images were superimposed on the 2D ToF intensity images, and the target visualization error (TVE)
was computed as quality measure. Color video data of the same organs were further used to assess the TVE of a
previously proposed marker-based registration method. The ToF-based registration showed promising accuracy
yielding a mean TVE of 2.5±1.1 mm compared to 0.7±0.4 mm with the marker-based approach. Furthermore,
the target registration error (TRE) was assessed to determine the anisotropy in the localization error of ToF
image data. The TRE was 8.9± 4.7 mm on average indicating a high localization error in the viewing direction
of the camera. Nevertheless, the young ToF technique may become a valuable means for intra-operative surface
acquisition. Future work should focus on the calibration of systematic distance errors.
Registration of multiple medical images commonly comprises the steps feature extraction, correspondences search and transformation computation. In this paper, we present a new method for a fast and pose independent search of correspondences using as features anatomical trees such as the bronchial system in the lungs or the vessel system in the liver. Our approach scores the similarities between the trees' nodes (bifurcations) taking into account both, topological properties extracted from their graph representations and anatomical properties extracted from the trees themselves. The node assignment maximizes the global similarity (sum of the scores of each pair of assigned nodes), assuring that the matches are distributed throughout the trees. Furthermore, the proposed method is able to deal with distortions in the data, such as noise, motion, artifacts, and problems associated with the extraction method, such as missing or false branches. According to an evaluation on swine lung data sets, the method requires less than one second on average to compute the matching and yields a high rate of correct matches compared to state of the art work.
KEYWORDS: Endoscopy, Video, 3D acquisition, Medical imaging, Visualization, Visual process modeling, Current controlled current source, Laparoscopy, 3D modeling, Natural surfaces
A growing number of applications in the field of computer-assisted laparoscopic interventions depend on accurate and fast 3D surface acquisition. The most commonly applied methods for 3D reconstruction of organ surfaces from 2D endoscopic images involve establishment of correspondences in image pairs to allow for computation of 3D point coordinates via triangulation. The popular feature-based approach for correspondence search applies a feature descriptor to compute high-dimensional feature vectors describing the characteristics of selected image points. Correspondences are established between image points with similar feature vectors. In a previous study, the performance of a large set of state-of-the art descriptors for the use in minimally invasive surgery was assessed. However, standard Phase Alternating Line (PAL) endoscopic images were utilized for this purpose. In this paper, we apply some of the best performing feature descriptors to in-vivo PAL endoscopic images as well as to High Definition Television (HDTV) endoscopic images of the same scene and show that the quality of the correspondences can be increased significantly when using high resolution images.
KEYWORDS: Particles, Bronchoscopy, Particle filters, Electromagnetism, Lung, Medical imaging, Visualization, Motion models, Visual process modeling, Current controlled current source
Although the field of a navigated bronchoscopy gains increasing attention in the literature, robust guidance in the presence of respiratory motion and electromagnetic noise remains challenging.
The robustness of a previously introduced motion compensation approach was increased by taking into account the already traveled trajectory of the instrument within the lung. To evaluate the performance of the method a virtual environment, which accounts for respiratory motion and electromagnetic noise was used. The simulation is based on a deformation field computed from human computed tomography data. According to the results, the proposed method outperforms the original method and is suitable for lung motion compensation during electromagnetically guided interventions.
The main challenges of Computed Tomography (CT)-guided organ puncture are the mental registration of the
medical imaging data with the patient anatomy, required when planning a trajectory, and the subsequent precise
insertion of a needle along it. An interventional telerobotic system, such as Robopsy, enables precise needle
insertion, however, in order to minimize procedure time and number of CT scans, this system should be driven
by an interface that is directly integrated with the medical imaging data. In this study we have developed and
evaluated such an interface that provides the user with a point-and-click functionality for specifying the desired
trajectory, segmenting the needle and automatically calculating the insertion parameters (angles and depth).
In order to highlight the advantages of such an interface, we compared robotic-assisted targeting using the old
interface (non-image-based) where the path planning was performed on the CT console and transferred manually
to the interface with the targeting procedure using the new interface (image-based). We found that the mean
procedure time (n=5) was 22±5 min (non-image-based) and 19±1 min (image-based) with a mean number of CT
scans of 6±1 (non-image-based) and 5±1 (image-based). Although the targeting experiments were performed
in gelatin with homogenous properties our results indicate that an image-based interface can reduce procedure
time as well as number of CT scans for percutaneous needle biopsies.
Lena Maier-Hein, Conor Walsh, Alexander Seitel, Nevan Hanumara, Jo-Anne Shepard, A. Franz, F. Pianka, Sascha Müller, Bruno Schmied, Alexander Slocum, Rajiv Gupta, Hans-Peter Meinzer
Computed tomography (CT) guided percutaneous punctures of the liver for cancer diagnosis and therapy (e.g.
tumor biopsy, radiofrequency ablation) are well-established procedures in clinical routine. One of the main
challenges related to these interventions is the accurate placement of the needle within the lesion. Several
navigation concepts have been introduced to compensate for organ shift and deformation in real-time, yet, the
operator error remains an important factor influencing the overall accuracy of the developed systems. The aim
of this study was to investigate whether the operator error and, thus, the overall insertion error of an existing
navigation system could be further reduced by replacing the user with the medical robot Robopsy. For this
purpose, we performed navigated needle insertions in a static abdominal phantom as well as in a respiratory
liver motion simulator and compared the human operator error with the targeting error performed by the robot.
According to the results, the Robopsy driven needle insertion system is able to more accurately align the needle
and insert it along its axis compared to a human operator. Integration of the robot into the current navigation
system could thus improve targeting accuracy in clinical use.
We compare two optical tracking systems with regard to their suitability for soft tissue navigation with fiducial
needles: The Polaris system with passive markers (Northern Digital Inc. (NDI); Waterloo, Ontario, Canada),
and the MicronTracker 2, model H40 (Claron Technology, Inc.; Toronto, Ontario, Canada). We introduce
appropriate tool designs and assess the tool tip tracking accuracy under typical clinical light conditions in a
sufficiently sized measurement volume. To assess the robustness of the tracking systems, we further evaluate
their sensitivity to illumination conditions as well as to the velocity and the orientation of a tracked tool. While
the Polaris system showed robust tracking accuracy under all conditions, the MicronTracker 2 was highly
sensitive to the examined factors.
We evaluate two core modules of a novel soft tissue navigation system. The system estimates the position of
a hidden target (e.g. a tumor) during a minimally invasive intervention from the location of a set of optically
tracked needle-shaped navigation aids which are placed in the vicinity of the target. The initial position of the
target relative to the navigation aids is obtained from a CT scan. The accuracy of the entire system depends on
(a) the accuracy for locating a set of navigation aids in a CT image, (b) the accuracy for determining the positions
of the navigation aids during the intervention by means of optical tracking, (c) the accuracy for tracking the
applicator (e.g. the biopsy needle), and (d) the accuracy of the real-time deformation model which continuously
computes the location of the initially determined target point from the current positions of the navigation aids.
In this paper, we focus on the first two aspects. We introduce the navigation aids we constructed for our
system and show that the needle tips can be tracked with submillimeter accuracy. Furthermore, we present and
evaluate three methods for registering a set of navigation aid models with a given CT image. The fully-automatic
algorithm outperforms both the manual method and the semi-automatic algorithm, yielding an average distance
of 0.27 ± 0.08 mm between the estimated needle tip position and the reference position.
In this paper, we evaluate the target position estimation accuracy of a novel soft tissue navigation system with a
custom-designed respiratory liver motion simulator. The system uses a real-time deformation model to estimate
the position of the target (e.g. a tumor) during a minimally invasive intervention from the location of a set of
optically tracked needle-shaped navigation aids which are placed in the vicinity of the target.
A respiratory liver motion simulator was developed to evaluate the performance of the system in-vitro. It
allows the mounting of an explanted liver which can be moved along the longitudinal axis of a corpus model to
simulate breathing motion. In order to assess the accuracy of our system we utilized an optically trackable tool
as target and estimated its position continuously from the current position of the navigation aids. Four different
transformation types were compared as base for the real-time deformation model: Rigid transformations, thinplate
splines, volume splines, and elastic body splines. The respective root-mean-square target position estimation
errors are 2.15 mm, 1.60 mm, 1.88 mm, and 1.92 mm averaged over a set of experiments obtained from a total
of six navigation aid configurations in two pig livers. The error is reduced by 76.3%, 82.4%, 79.3%, and 78.8%,
respectively, compared to the case when no deformation model is applied, i.e., a constant organ position is
assumed throughout the breathing cycle.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.