One of the fundamental components in all Image Guided Surgery (IGS) applications is a method for presenting
information to the surgeon in a simple, effective manner. This paper describes the first steps in our new
Augmented Reality (AR) information delivery program. The system makes use of new "off the shelf" AR glasses
that are both light-weight and unobtrusive, with adequate resolution for many IGS applications. Our first
application is perioperative planning of minimally invasive robot-assisted cardiac surgery. In this procedure,
a combination of tracking technologies and intraoperative ultrasound is used to map the migration of cardiac
targets prior to selection of port locations for trocars that enter the chest. The AR glasses will then be used to
present this heart migration data to the surgeon, overlaid onto the patients chest. The current paper describes
the calibration process for the AR glasses, their integration into our IGS framework for minimally invasive robotic
cardiac surgery, and preliminary validation of the system. Validation results indicate a mean 3D triangulation
error of 2.9 ± 3.3mm, 2D projection error of 2.1 ± 2.1 pixels, and Normalized Stereo Calibration Error of 3.3.
Image-guided interventions rely on the common assumption that pre-operative information can depict intraoperative
morphology with sufficient accuracy. Nevertheless, in the context of minimally invasive cardiac therapy
delivery, this assumption loses ground; the heart is a soft-tissue organ prone to changes induced during access to
the heart and especially intracardiac targets. In addition to its clinical value for cardiac interventional guidance
and assistance with the image- and model-to-patient registration, here we show how ultrasound imaging may be
used to estimate changes in the heart position and morphology of structures of interest at different stages in the
procedure. Using a magnetically tracked 2D transesophageal echocardiography transducer, we acquired in vivo
images of the heart at different stages during the procedural workflow of common minimally invasive cardiac
procedures, including robot-assisted coronary artery bypass grafting, mitral valve replacement/repair, or modelenhanced
US-guided intracardiac interventions, all in the coordinate system of the tracking system. Anatomical
features of interest (mitral and aortic valves) used to register the pre-operative anatomical models to the intraoperative
coordinate frame were identified from each dataset. This information allowed us to identify the global
position of the heart and also characterize the valvular structures at various peri-operative stages, in terms of
their orientation, size, and geometry. Based on these results, we can estimate the differences between the preand
intra-operative anatomical features, their effect on the model-to-subject registration, and also identify the
need to update or optimize any pre-operative surgical plan to better suit the intra-operative procedure workflow.
Trans-esophageal echocardiography (TEE) is a standard component of patient monitoring during most cardiac
surgeries. In recent years magnetic tracking systems (MTS) have become sufficiently robust to function effectively
in appropriately structured operating room environments. The ability to track a conventional multiplanar 2D
TEE transducer in 3D space offers incredible potential by greatly expanding the cumulative field of view of cardiac
anatomy beyond the limited field of view provided by 2D and 3D TEE technology. However, there is currently
no TEE probe manufactured with MTS technology embedded in the transducer, which means sensors must be
attached to the outer surface of the TEE. This leads to potential safety issues for patients, as well as potential
damage to the sensor during procedures. This paper presents a standard 2D TEE probe fully integrated with
MTS technology. The system is evaluated in an environment free of magnetic and electromagnetic disturbances,
as well as a clinical operating room in the presence of a da Vinci robotic system. Our first integrated TEE
device is currently being used in animal studies for virtual reality-enhanced ultrasound guidance of intracardiac
surgeries, while the "second generation" TEE is in use in a clinical operating room as part of a project to
measure perioperative heart shift and optimal port placement for robotic cardiac surgery. We demonstrate
excellent system accuracy for both applications.
Peripheral nerve block treatments are ubiquitous in hospitals and pain clinics worldwide. State of the art
techniques use ultrasound (US) guidance and/or electrical stimulation to verify needle tip location. However,
problems such as needle-US beam alignment, poor echogenicity of block needles and US beam thickness can
make it difficult for the anesthetist to know the exact needle tip location. Inaccurate therapy delivery raises
obvious safety and efficacy issues. We have developed and evaluated a needle guidance system that makes use
of a magnetic tracking system (MTS) to provide an augmented reality (AR) guidance platform to accurately
localize the needle tip as well as its projected trajectory. Five anesthetists and five novices performed simulated
nerve block deliveries in a polyvinyl alcohol phantom to compare needle guidance under US alone to US placed in
our AR environment. Our phantom study demonstrated a decrease in targeting attempts, decrease in contacting
of critical structures, and an increase in accuracy of 0.68 mm compared to 1.34mm RMS in US guidance alone.
Currently, the MTS uses 18 and 21 gauge hypodermic needles with a 5 degree of freedom sensor located at the
needle tip. These needles can only be sterilized using an ethylene oxide process. In the interest of providing
clinicians with a simple and efficient guidance system, we also evaluated attaching the sensor at the needle hub as
a simple clip-on device. To do this, we simultaneously performed a needle bending study to assess the reliability
of a hub-based sensor.
The development of an augmented reality environment that combines laparoscopic video and ultrasound imaging
for image-guided minimally invasive abdominal surgical procedures, such as partial nephrectomy and radical
prostatectomy, is an ongoing project in our laboratory. Our system overlays magnetically tracked ultrasound
images onto endoscopic video to create a more intuitive visualization for mapping lesions intraoperatively and to
give the ultrasound image context in 3D space. By presenting data in a common environment, this system will
allow surgeons to visualize the multimodality information without having to switch between different images.
A stereoscopic laparoscope from Visionsense Limited enhances our current system by providing surgeons with
additional visual information through improved depth perception. In this paper, we develop and validate a
calibration method that determines the transformation between the images from the stereoscopic laparoscope
and the 3D locations of structures represented by a tracked laparoscopic ultrasound probe. We first calibrate
the laparoscope with a checkerboard pattern and measure how accurate the transformation from image space
to tracking space is. We then perform a target localization task using our fused environment. Our initial
experience has demonstrated an RMS registration accuracy in 3D of 2.21mm for the laparoscope and 1.16mm for
the ultrasound in a working volume of 0.125m3, indicating that magnetically tracked stereoscopic laparoscope
and ultrasound images may be appropriately combined using magnetic tracking as long as steps are taken to
ensure that the magnetic field generated by the system is not distorted by surrounding objects close to the
working volume.
KEYWORDS: Heart, Endoscopy, In vitro testing, Autoregressive models, Visualization, Augmented reality, Data modeling, Magnetic tracking, Real time imaging, Transducers
Catheter-driven cardiac interventions have emerged in response to the need of reducing invasiveness associated
with the traditional cut-and-sew techniques. Catheter manipulation is traditionally performed under real-time
fluoroscopy imaging, resulting in an overall trade-off of procedure invasiveness for radiation exposure of both
the patient and clinical staff. Our approach to reducing and potentially eliminating the use of flouroscopy
in the operating room entails the use of multi-modality imaging and magnetic tracking technologies, wrapped
together into an augmented reality environment for enhanced intra-procedure visualization and guidance. Here
we performed an in vitro study in which a catheter was guided to specific targets located on the endocardial atrial
surface of a beating heart phantom. "Therapy delivery" was modeled in the context of a blinded procedure,
mimicking a beating heart, intracardiac intervention. The users navigated the tip of a magnetically tracked
Freezor 5 CRYOCATH catheter to the specified targets. Procedure accuracy was determined as the distance
between the tracked catheter tip and the tracked surgical target at the time of contact, and it was assessed under
three different guidance modalities: endoscopic, augmented reality, and ultrasound image guidance. The overall
RMS targeting accuracy achieved under augmented reality guidance averaged to 1.1 mm. This guidance modality
shows significant improvements in both procedure accuracy and duration over ultrasound image guidance alone,
while maintianing an overall targeting accuracy comparable to that achieved under endoscopic guidance.
Ultrasound is garnering significant interest as an imaging modality for surgical guidance, due to its affordability,
real-time temporal resolution and ease of integration into the operating room. Minimally-invasive intracardiac
surgery performed on the beating-heart prevents direct vision of the surgical target, and procedures such as
mitral valve replacement and atrial septal defect closure would benefit from intraoperative ultrasound imaging.
We propose that placing 4D ultrasound within an augmented reality environment, along with a patient-specific
cardiac model and virtual representations of tracked surgical tools, will create a visually intuitive platform with
sufficient image information to safely and accurately repair tissue within the beating heart. However, the quality
of the imaging parameters, spatial calibration, temporal calibration and ECG-gating must be well characterized
before any 4D ultrasound system can be used clinically to guide the treatment of moving structures. In this paper,
we describe a comprehensive accuracy assessment framework that can be used to evaluate the performance of 4D
ultrasound systems while imaging moving targets. We image a dynamic phantom that is comprised of a simple
robot and a tracked phantom to which point-source, distance and spherical objects of known construction can be
attached. We also follow our protocol to evaluate 4D ultrasound images generated in real-time by reconstructing
ECG-gated 2D ultrasound images acquired from a tracked multiplanar transesophageal probe. Likewise, our
evaluation framework allows any type of 4D ultrasound to be quantitatively assessed.
KEYWORDS: Heart, 3D modeling, Surgery, 3D image processing, Image segmentation, Virtual reality, Visualization, Magnetic resonance imaging, Image registration, In vivo imaging
As part of an ongoing theme in our laboratory on reducing morbidity during minimally-invasive intracardiac
procedures, we developed a computer-assisted intervention system that provides safe access inside the beating
heart and sufficient visualization to deliver therapy to intracardiac targets while maintaining the efficacy of the
procedure. Integrating pre-operative information, 2D trans-esophageal ultrasound for real-time intra-operative
imaging, and surgical tool tracking using the NDI Aurora magnetic tracking system in an augmented virtual
environment, our system allows the surgeons to navigate instruments inside the heart in spite of the lack of
direct target visualization. This work focuses on further enhancing intracardiac visualization and navigation by
supplying the surgeons with detailed 3D dynamic cardiac models constructed from high-resolution pre-operative
MR data and overlaid onto the intra-operative imaging environment. Here we report our experience during an in
vivo porcine study. A feature-based registration technique previously explored and validated in our laboratory
was employed for the pre-operative to intra-operative mapping. This registration method is suitable for in
vivo interventional applications as it involves the selection of easily identifiable landmarks, while ensuring a good
alignment of the pre-operative and intra-operative surgical targets. The resulting augmented reality environment
fuses the pre-operative cardiac model with the intra-operative real-time US images with approximately 5 mm
accuracy for structures located in the vicinity of the valvular region. Therefore, we strongly believe that our
augmented virtual environment significantly enhances intracardiac navigation of surgical instruments, while on-target
detailed manipulations are performed under real-time US guidance.
A 2D ultrasound enhanced virtual reality surgical guidance system has been under development for some time in
our lab. The new surgical guidance platform has been shown to be effective in both the laboratory and clinical
settings, however, the accuracy of the tracked 2D ultrasound has not been investigated in detail in terms of the
applications for which we intend to use it (i.e., mitral valve replacement and atrial septal defect closure). This
work focuses on the development of an accuracy assessment protocol specific to the assessment of the calibration
methods used to determine the rigid transformation between the ultrasound image and the tracked sensor.
Specifically, we test a Z-bar phantom calibration method and a phantomless calibration method and compared
the accuracy of tracking ultrasound images from neuro, transesophageal, intracardiac and laparoscopic ultrasound
transducers. This work provides a fundamental quantitative description of the image-guided accuracy that can
be obtained with this new surgical guidance system.
Minimally invasive techniques for use inside the beating heart, such as mitral valve replacement and septal defect
repair, are the focus of this work. Traditional techniques for these procedures require an open chest approach
and a cardiopulmonary bypass machine. New techniques using port access and a combined surgical guidance tool
that includes an overlaid two-dimensional ultrasound image in a virtual reality environment are being developed.
To test this technique, a cardiac phantom was developed to simulate the anatomy. The phantom consists of an
acrylic box filled with a 7% glycerol solution with ultrasound properties similar to human tissue. Plate inserts
mounted in the box simulate the physical anatomy. An accuracy assessment was completed to evaluate the
performance of the system.
Using the cardiac phantom, a 2mm diameter glass toroid was attached to a vertical plate as the target
location. An elastic material was placed between the target and plate to simulate the target lying on a soft tissue
structure. The target was measured using an independent measurement system and was represented as a sphere
in the virtual reality system. The goal was to test the ability of a user to probe the target using three guidance
methods: (i) 2D ultrasound only, (ii) virtual reality only and (iii) ultrasound enhanced virtual reality. Three
users attempted the task three times each for each method. An independent measurement system was used
to validate the measurement. The ultrasound imaging alone was poor in locating the target (5.42 mm RMS)
while the other methods proved to be significantly better (1.02 mm RMS and 1.47 mm RMS respectively). The
ultrasound enhancement is expected to be more useful in a dynamic environment where the system registration may be disturbed.
Clinical research has been rapidly evolving towards the development of less invasive surgical procedures. We
recently embarked on a project to improve intracardiac beating heart interventions. Our novel approach employs
new surgical technologies and support from image-guidance via pre-operative and intra-operative imaging (i.e.
two-dimensional echocardiography) to substitute for direct vision. Our goal was to develop a versatile system
that allowed for safe cardiac port access, and provide sufficient image-guidance with the aid of a virtual reality
environment to substitute for the absence of direct vision, while delivering quality therapy to the target. Specific targets included the repair and replacement of heart valves and the repair of septal defects. The ultimate
objective was to duplicate the success rate of conventional open-heart surgery, but to do so via a small incision,
and to evaluate the efficacy of the procedure as it is performed. This paper describes the software and hardware
components, along with the methodology for performing mitral valve replacement as one example of this
approach, using ultrasound and virtual tool models to position and fasten the valve in place.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.