Including uncertainty information in the assessment of a segmentation of pathologic structures on medical images, offers the potential to increase trust into deep learning algorithms for the analysis of medical imaging. Here, we examine options to extract uncertainty information from deep learning segmentation models and the influence of the choice of cost functions on these uncertainty measures. To this end we train conventional UNets without dropout, deep UNet ensembles, and Monte-Carlo (MC) dropout UNets to segment lung nodules on low dose CT using either soft Dice or weighted categorical cross-entropy (wcc) as loss functions. We extract voxel-wise uncertainty information from UNet models based on softmax maximum probability and from deep ensembles and MC dropout UNets using mean voxel-wise entropy. Upon visual assessment, areas of high uncertainty are localized in the periphery of segmentations and are in good agreement with incorrectly labelled voxels. Furthermore, we evaluate how well uncertainty measures correlate with segmentation quality (Dice score). Mean uncertainty over the segmented region (Ulabelled) derived from conventional UNet models does not show a strong quantitative relationship with the Dice score (Spearman correlation coefficient of -0.45 for the soft Dice vs -0.64 for the wcc model respectively). By comparison, image-level uncertainty measures derived from soft Dice as well as wcc MC UNet and deep UNet ensemble models correlate well with the Dice score. In conclusion, using uncertainty information offers ways to assess segmentation quality fully automatically without access to ground truth. Models trained using weighted categorical cross-entropy offer more meaningful uncertainty information on a voxel-level.
Radiomics has shown promising results in several medical studies, yet it suffers from a limited discrimination and informative capability as well as a high variation and correlation with the tomographic scanner types, pixel spacing, acquisition protocol, and reconstruction parameters. We propose and compare two methods to transform quantitative image features in order to improve their stability across varying image acquisition parameters while preserving the texture discrimination abilities. In this way, variations in extracted features are representative of true physiopathological tissue changes in the scanned patients. A first approach is based on a two-layer neural network that can learn a nonlinear standardization transformation of various types of features including handcrafted and deep features. Second, domain adversarial training is explored to increase the invariance of the transformed features to the scanner of origin. The generalization of the proposed approach to unseen textures and unseen scanners is demonstrated by a set of experiments using a publicly available computed tomography texture phantom dataset scanned with various imaging devices and parameters.
Radiomics has shown promising results in several medical studies, yet it suffers from a limited discrimination and informative capability as well as a high variation and correlation with the tomographic scanner types, CT (Computer Tomography) scanner producers, pixel spacing, acquisition protocol and reconstruction parameters. This paper introduces a new method to transform image features in order to improve their stability across scanner producers and scanner models. This method is based on a two-layer neural network that can learn a non-linear standardization transformation of various types of features including hand-crafted and deep features. A publicly available database of phantom images with ground truth is used where the same physical phantom was scanned on 17 different CT scanners. In this setting, variations in extracted features are representative of true physio-pathological tissue changes in the scanned patients, so harmonized between scanner producers and models. The recent success of radiomics studies has often been restricted to relatively controlled environments. In order allow for comparing data of several hospitals produced with a larger variety of scanner producers and models as well as with several protocols, features standardization seems necessary to keep results comparable.
Diagnosing Interstitial Lung Diseases (ILD) is a difficult task. It requires experienced chest radiologists that may not be available in less-specialized health centers. Moreover, a correct diagnosis is needed to decide for an appropriate treatment and prognostic. In this paper, we focus on the classification of 3 common subtypes of ILDs: Usual Interstitial Pneumonia (UIP), Non-Specific Interstitial Pneumonia (NSIP) and Chronic Hypersensitivity Pneumonitis (CHP). We propose a graph model of the lungs built from a large dataset. The structure of the graph is inspired from medical knowledge of disease predominance, where the nodes correspond to 24 distinct regions obtained from lateral, anterior-posterior and vertical splits of the images. The adjacency matrix is built from distances between intensity distributions of distinct regions. Graphs models are interpretable and were successfully used in neuroimaging. However, to the best of our knowledge, this is the first attempt to use a graph model of the lungs for classifying ILDs. In the particular case of ILDs, graph methods are relevant for the following reasons. In order to differentiate between the subtypes, not only the types of local patterns of the disease are important but also their anatomical location. Therefore, we hypothesize that the comparison between regional distributions of Hounsfield Unit (HU) values is relevant to discriminate between the considered ILD subtypes. For instance, typical UIP shows a spatial predominance of reticular abnormalities and honeycombing in the peripheral regions of the lung bases. Therefore, we expect a marked difference of HU distributions between the central and peripheral regions of the lung bases. Moreover, the construction of the graph leads to an interpretable patient descriptor. The descriptor led to encouraging area under the Receiver Operating Characteristic (ROC) curve in 0.6-0.8 for one-versus-one classification configurations, which also showed to outperform feature sets based on a simple concatenation of regional HU distributions.
KEYWORDS: Breast cancer, Performance modeling, Tumors, Magnetic resonance imaging, Data modeling, Wavelets, Statistical modeling, Image filtering, Statistical analysis, Breast
Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is sensitive but not specific to determining treatment response in early stage triple-negative breast cancer (TNBC) patients. We propose an efficient computerized technique for assessing treatment response, specifically the residual tumor (RT) status and pathological complete response (pCR), in response to neoadjuvant chemotherapy. The proposed approach is based on Riesz wavelet analysis of pharmacokinetic maps derived from noninvasive DCE-MRI scans, obtained before and after treatment. We compared the performance of Riesz features with the traditional gray level co-occurrence matrices and a comprehensive characterization of the lesion that includes a wide range of quantitative features (e.g., shape and boundary). We investigated a set of predictive models (∼96) incorporating distinct combinations of quantitative characterizations and statistical models at different time points of the treatment and some area under the receiver operating characteristic curve (AUC) values we reported are above 0.8. The most efficient models are based on first-order statistics and Riesz wavelets, which predicted RT with an AUC value of 0.85 and pCR with an AUC value of 0.83, improving results reported in a previous study by ∼13%. Our findings suggest that Riesz texture analysis of TNBC lesions can be considered a potential framework for optimizing TNBC patient care.
Pulmonary embolism (PE) affects up to 600,000 patients and contributes to at least 100,000 deaths every year in the United States alone. Diagnosis of PE can be difficult as most symptoms are unspecific and early diagnosis is essential for successful treatment. Computed Tomography (CT) images can show morphological anomalies that suggest the existence of PE. Various image-based procedures have been proposed for improving computer-aided diagnosis of PE. We propose a novel method for detecting PE based on localized vessel-based features computed in Dual Energy CT (DECT) images. DECT provides 4D data indexed by the three spatial coordinates and the energy level. The proposed features encode the variation of the Hounsfield Units across the different levels and the CT attenuation related to the amount of iodine contrast in each vessel. A local classification of the vessels is obtained through the classification of these features. Moreover, the localization of the vessel in the lung provides better comparison between patients. Results show that the simple features designed are able to classify pulmonary embolism patients with an AUC (area under the receiver operating curve) of 0.71 on a lobe basis. Prior segmentation of the lung lobes is not necessary because an automatic atlas-based segmentation obtains similar AUC levels (0.65) for the same dataset. The automatic atlas reaches 0.80 AUC in a larger dataset with more control cases.
Medical images contain a large amount of visual information about structures and anomalies in the human body. To make sense of this information, human interpretation is often essential. On the other hand, computer-based approaches can exploit information contained in the images by numerically measuring and quantifying specific visual features. Annotation of organs and other anatomical regions is an important step before computing numerical features on medical images. In this paper, a texture-based organ classification algorithm is presented, which can be used to reduce the time required for annotating medical images. The texture of organs is analyzed using a combination of state-of-the-art techniques: the Riesz transform and a bag of meaningful visual words. The effect of a meaningfulness transformation in the visual word space yields two important advantages that can be seen in the results. The number of descriptors is enormously reduced down to 10% of the original size, whereas classification accuracy is improved by up to 25% with respect to the baseline approach.
Distinct texture classes are often sharing several visual concepts. Texture instances from different classes are sharing regions in the feature hyperspace, which results in ill-defined classification configurations. In this work, we detect rotation-covariant visual concepts using steerable Riesz wavelets and bags of visual words. In a first step, K-means clustering is used to detect visual concepts in the hyperspace of the energies of steerable Riesz wavelets. The coordinates of the clusters are used to construct templates from linear combinations of the Riesz components that are corresponding to visual concepts. The visualization of these templates allows verifying the relevance of the concepts modeled. Then, the local orientations of each template are optimized to maximize their response, which is carried out analytically and can still be expressed as a linear combination of the initial steerable Riesz templates. The texture classes are learned in the feature space composed of the concatenation of the maximum responses of each visual concept using support vector machines. An experimental evaluation using the Outex TC 00010 test suite allowed a classification accuracy of 97.5%, which demonstrates the feasibility of the proposed approach. An optimal number K = 20 of clusters is required to model the visual concepts, which was found to be fewer than the number of classes. This shows that higher-level classes are sharing low-level visual concepts. The importance of rotation-covariant visual concept modeling is highlighted by allowing an absolute gain of more than 30% in accuracy. The visual concepts are modeling the local organization of directions at various scales, which is in accordance with the bottom{up visual information processing sequence of the primal sketch in Marr's theory on vision.
Volumetric medical images contain an enormous amount of visual information that can discourage the exhaustive use of local descriptors for image analysis, comparison and retrieval. Distinctive features and patterns that need to be analyzed for finding diseases are most often local or regional, often in only very small parts of the image. Separating the large amount of image data that might contain little important information is an important task as it could reduce the current information overload of physicians and make clinical work more efficient. In this paper a novel method for detecting key-regions is introduced as a way of extending the concept of keypoints often used in 2D image analysis. In this way also computation is reduced as important visual features are only extracted from the detected key regions. The region detection method is integrated into a platform-independent, web-based graphical interface for medical image visualization and retrieval in three dimensions. This web-based interface makes it easy to deploy on existing infrastructures in both small and large-scale clinical environments. By including the region detection method into the interface, manual annotation is reduced and time is saved, making it possible to integrate the presented interface and methods into clinical routine and workflows, analyzing image data at a large scale.
Comparing several series of images is not always easy as the corresponding slices often need
to be selected manually. In times where series contain an ever-increasing number of slices this
can mean manual work when moving several series to the corresponding slice. Particularly two
situations were identified in this context: (1) patients with a large number of image series over
time (such as patients with cancers that are monitored) frequently need to compare the series,
for example to compare tumor growth over time. Manually adapting two series is possible but
with four or more series this can mean loosing time. Having automatically the closest slice
by comparing visual similarity also in older series with differing slice thickness and inter slice
distance can save time and synchronize the viewing instantly. (2) analyzing visually similar
image series of several patients can profit from being viewed in a synchronized way to compare
the cases, so when sliding through the slices in one volume, the corresponding slices in the other
volumes are shown. This application could be employed after content-based 3D image retrieval
has found similar series, for example. Synchronized viewing can help finding or confirming the
most relevant cases quickly.
To allow for synchronized viewing of several image volumes, the test image series are first
registered applying affine transformation for the global registration of images followed by diffeomorphic
image registration. Then corresponding slices in the two volumes are estimated based
on a visual similarity. Once the registration is finished, the user can subsequently move inside
the slices of one volume (reference volume) and can view the corresponding slices in the other
volumes. These corresponding slices are obtained after a correspondence match in the registration
procedure. These volumes are synchronized in that the slice closest to the original reference
volume is shown even when the slice thicknesses or inter slice distances differ, and this is automatically
done by comparing the visual image content of the slices. The tool has the potential to
help in a variety of situations and it is currently being made available as a plugin for the popular
Osirix image viewer.
Segmentation of the various parts of the brain is a challenging area in medical imaging and it is a prerequisite
for many image analysis tasks useful for clinical research. Advances have been made in generating brain image
templates that can be registered to automatically segment regions of interest in the human brain. However, these
methods may fail with some subjects if there is a significant shape distortion or difference from the proposed
models. This is also the case of newborns, where the developing brain strongly differs from adult magnetic
resonance imaging (MRI) templates.
In this article, a texture-based cerebellum segmentation method is described. The algorithm presented does
not use any prior spatial knowledge to segment the MRI images. Instead, the system learns the texture features
by means of a multi-scale filtering and visual words feature aggregation. Visual words are a commonly used
technique in image retrieval. Instead of using visual features directly, the features of specific regions are modeled
(clustered) into groups of discriminative features. This means that the final feature space can be reduced in size
and also that the visual words in local regions are really discriminative for the given data set. The system is
currently trained and tested with a dataset of 18 adult brain MRIs. An extension to the use with newborn brain
images is being foreseen as this could highlight the advantages of the proposed technique.
Results show that the use of texture features can be valuable for the task described and can lead to good
results. The use of visual words can potentially improve robustness of existing shape-based techniques for cases
with significant shape distortion or other differences from the models. As the visual words based techniques are
not assuming any prior knowledge such techniques could be used for other types of segmentations as well using
a large variety of basic visual features.
Fractures are common injuries, some complicated fractures may require a surgical intervention. When such an
operation is planned it can be beneficial to have access to similar past cases including follow ups to compare, which
method might be the most adapted one in a particular situation. At the orthopaedic service of the University
hospitals of Geneva a database of past cases including pre- and post-operative images and case descriptions has
been created over the past years with the goal to support clinical decision making.
Images play an important role in the decision making process and the judgment of a fracture, but visual
image content is currently not directly accessible for search. At the moment, search is mainly via a classification
system of the fractures or in the patient record itself only by patient ID. In this paper we propose a solution
that combines visual information from several images in a case to calculate similarity between cases and allow
thus an access to visually similar cases. Such a system can complement the text- or classification-based search
that has been used so far.
In a preliminary study, we used pixel-grid-based salient-point features to build a first prototype of case-based
visual retrieval of fracture cases. Cases belonging to different fracture classes were beforehand often confused
due to the similar bone structures in the various images. In this article, a multi-scale approach is used in
order to perform similarity measures at both large and small scales. When compared to the first prototype, the
introduction of scale and spatial information allowed improving the performance of the system. Cases containing
similar bone structures but with dissimilar fractures are generally ranked lower whereas more relevant cases are
returned. The system can thus be expected to perform sufficiently well for use in clinical practice and particularly
for teaching.
Images are an integral part of medical practice for diagnosis, treatment planning and teaching. Image retrieval
has gained in importance mainly as a research domain over the past 20 years. Both textual and visual retrieval of
images are essential. In the process of mobile devices becoming reliable and having a functionality equaling that
of formerly desktop clients, mobile computing has gained ground and many applications have been explored. This
creates a new field of mobile information search & access and in this context images can play an important role
as they often allow understanding complex scenarios much quicker and easier than free text. Mobile information
retrieval in general has skyrocketed over the past year with many new applications and tools being developed
and all sorts of interfaces being adapted to mobile clients.
This article describes constraints of an information retrieval system including visual and textual information
retrieval from the medical literature of BioMedCentral and of the RSNA journals Radiology and Radiographics.
Solutions for mobile data access with an example on an iPhone in a web-based environment are presented
as iPhones are frequently used and the operating system is bound to become the most frequent smartphone
operating system in 2011. A web-based scenario was chosen to allow for a use by other smart phone platforms
such as Android as well. Constraints of small screens and navigation with touch screens are taken into account
in the development of the application. A hybrid choice had to be taken to allow for taking pictures with the
cell phone camera and upload them for visual similarity search as most producers of smart phones block this
functionality to web applications.
Mobile information access and in particular access to images can be surprisingly efficient and effective on
smaller screens. Images can be read on screen much faster and relevance of documents can be identified quickly
through the use of images contained in the text. Problems with the many, often incompatible mobile platforms
were discovered and are listed in the text. Mobile information access is a quickly growing domain and the
constraints of mobile access also need to be taken into account for image retrieval. The demonstrated access to
the medical literature is most relevant as the medical literature and their images are clearly the largest knowledge
source in the medical field.
The interpretation of high-resolution computed tomography (HRCT) images of the chest showing disorders of the
lung tissue associated with interstitial lung diseases (ILDs) is time-consuming and requires experience. Whereas
automatic detection and quantification of the lung tissue patterns showed promising results in several studies, its
aid for the clinicians is limited to the challenge of image interpretation, letting the radiologists with the problem
of the final histological diagnosis. Complementary to lung tissue categorization, providing visually similar cases
using content-based image retrieval (CBIR) is in line with the clinical workflow of the radiologists.
In a preliminary study, a Euclidean distance based on volume percentages of five lung tissue types was used
as inter-case distance for CBIR. The latter showed the feasibility of retrieving similar histological diagnoses
of ILD based on visual content, although no localization information was used for CBIR. However, to retrieve
and show similar images with pathology appearing at a particular lung position was not possible. In this work,
a 3D localization system based on lung anatomy is used to localize low-level features used for CBIR. When
compared to our previous study, the introduction of localization features allows improving early precision for
some histological diagnoses, especially when the region of appearance of lung tissue disorders is important.
We compare five common classifier families in their ability to categorize six lung tissue patterns in high-resolution
computed tomography (HRCT) images of patients affected with interstitial lung diseases (ILD) but also normal
tissue. The evaluated classifiers are Naive Bayes, k-Nearest Neighbor (k-NN), J48 decision trees, Multi-Layer
Perceptron (MLP) and Support Vector Machines (SVM). The dataset used contains 843 regions of interest (ROI)
of healthy and five pathologic lung tissue patterns identified by two radiologists at the University Hospitals of
Geneva. Correlation of the feature space composed of 39 texture attributes is studied. A grid search for optimal
parameters is carried out for each classifier family. Two complementary metrics are used to characterize the
performances of classification. Those are based on McNemar's statistical tests and global accuracy. SVM
reached best values for each metric and allowed a mean correct prediction rate of 87.9% with high class-specific
precision on testing sets of 423 ROIs.
Interstitial lung diseases (ILDs) are a relatively heterogeneous group of around 150 illnesses with often very
unspecific symptoms. The most complete imaging method for the characterisation of ILDs is the high-resolution
computed tomography (HRCT) of the chest but a correct interpretation of these images is difficult even for
specialists as many diseases are rare and thus little experience exists. Moreover, interpreting HRCT images
requires knowledge of the context defined by clinical data of the studied case. A computerised diagnostic aid tool
based on HRCT images with associated medical data to retrieve similar cases of ILDs from a dedicated database
can bring quick and precious information for example for emergency radiologists. The experience from a pilot
project highlighted the need for detailed database containing high-quality annotations in addition to clinical
data.
The state of the art is studied to identify requirements for image-based diagnostic aid for interstitial lung
disease with secondary data integration. The data acquisition steps are detailed. The selection of the most
relevant clinical parameters is done in collaboration with lung specialists from current literature, along with
knowledge bases of computer-based diagnostic decision support systems. In order to perform high-quality
annotations of the interstitial lung tissue in the HRCT images an annotation software and its own file format
is implemented for DICOM images. A multimedia database is implemented to store ILD cases with clinical
data and annotated image series. Cases from the University & University Hospitals of Geneva (HUG) are
retrospectively and prospectively collected to populate the database. Currently, 59 cases with certified diagnosis
and their clinical parameters are stored in the database as well as 254 image series of which 26 have their regions
of interest annotated.
The available data was used to test primary visual features for the classification of lung tissue patterns. These
features show good discriminative properties for the separation of five classes of visual observations.
This article describes the use of a medical image retrieval system on a database of 16'000 fractures, selected from
surgical routine over several years. Image retrieval has been a very active domain of research for several years.
It was frequently proposed for the medical domain, but only few running systems were ever tested in clinical
routine. For the planning of surgical interventions after fractures, x-ray images play an important role. The
fractures are classified according to exact fracture location, plus whether and to which degree the fracture is
damaging articulations to see how complicated a reparation will be. Several classification systems for fractures
exist and the classification plus the experience of the surgeon lead in the end to the choice of surgical technique
(screw, metal plate, ...). This choice is strongly influenced by the experience and knowledge of the surgeons with
respect to a certain technique. Goal of this article is to describe a prototype that supplies similar cases to an
example to help treatment planning and find the most appropriate technique for a surgical intervention.
Our database contains over 16'000 fracture images before and after a surgical intervention. We use an image
retrieval system (GNU Image Finding Tool, GIFT) to find cases/images similar to an example case currently
under observation. Problems encountered are varying illumination of images as well as strong anatomic differences
between patients. Regions of interest are usually small and the retrieval system needs to focus on this region.
Results show that GIFT is capable of supplying similar cases, particularly when using relevance feedback, on
such a large database. Usual image retrieval is based on a single image as search target but for this application
we have to select images by case as similar cases need to be found and not images. A few false positive cases
often remain in the results but they can be sorted out quickly by the surgeons.
Image retrieval can well be used for the planning of operations by supplying similar cases. A variety of
challenges has been identified and partly solved (varying luminosity, small region of interested, case-based instead
of image-based). This article mainly presents a case study to identify potential benefits and problems. Several
steps for improving the system have been identified as well and will be described at the end of the paper.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.