1.

Introduction

Spectral imaging can be used to collect spectral information from a sample, and different imaging systems and configurations are available for variety of applications and fields, which include remote sensing,^1,2 quality assessment of agrofood products,^3,4 and biomedical and forensic imaging.^5–9 Normal, multispectral, hyperspectral, and ultraspectral imaging are some of the common terms used in spectral imaging. The main differences between them in the above mentioned order are with respect to a greater number of wavelength bands and higher precision. Two classification criteria are presented in Table 1.^1,10 Both criteria use the number of wavelength bands as one of the defining parameters. However, Fresse et al.¹⁰ used precision, whereas Puschell¹ used resolution. By using the number of wavelength bands, a common parameter between these two definitions, the latter makes it easier for one to classify a system as a hyperspectral (HS) or ultraspectral imager. The first definition which is stricter is used to define the type of spectral imaging that is employed in the pushbroom imager presented in this paper.

Table 1

Classification of spectral imaging.

Hyperspectral imaging (HSI) and ultraspectral imaging give more detailed spectral signatures for identification purposes with higher accuracy by enabling the detection of more wavelength bands along with an increase in spectral range, precision, and resolution.

Similar to other conventional spectral imaging, HSI requires a component such as a spectrograph to spectrally split or filter the incoming light from the sample. Thereafter, the light needs to be captured by a sensor. There are four main types of imager, namely spatial scanning, spectral scanning, spatiospectral scanning, and snapshot imagers. Each imager has its own advantages and constraints and can be used for specific objective or applications. They can be used to achieve the same type of information, known as a datacube. A datacube stores the intensity value in three dimensions (spatial-spatial-spectral). The value in each voxel [similar to pixels in a two-dimensional (2-D) dataset] of a datacube indicates the intensity of a specific wavelength from one spatial point in a 2-D sample.¹¹

A spatial-scanning imager usually uses a dispersive element such as a prism-grating-prism assembly in a spectrograph¹² to separate the incoming spectrum so that information of the constituent wavelength bands can be detected by the detector array. The conventional point-scanning spectrometer records the spectrum of a spatial point in each scan to give one-dimensional (1-D) spectral information. By repeating the scan across multiple points in a 2-D area (whiskbroom imager), a datacube can be formed. Spatial scanning can be done using a 2-D stage to move the sample or using a microelectromechanical system scanner to direct the point illumination to different parts of the sample.¹³ With a large sample, the data acquisition time can be long as scanning needs to be repeated for each point in the sample. A faster alternative in this category is to use a line-scanning or pushbroom imager.^14,15 In each scan, a line-scanning imager is able to capture the individual spectrum of every point across a line of the sample. The light from this line of the sample is dispersed into different wavelengths onto the 2-D detector. Thus, the image on the detector has one spatial and one spectral dimension. The scan is repeated by relative motion perpendicular to the detector’s line of view (LOV) of the sample. Line scanning is repeated in one dimension by going to a subsequent row, compared to a 2-D point scanning. This reduces the acquisition time in line scanning by a factor equal to the number of rows, which can be significant.

The development of acousto-optic tunable filters^16,17 and liquid crystal tunable filters^18,19 makes a spectral-scanning imager possible. These filters can be electronically tuned to selectively choose a wavelength band to be transmitted and be imaged by the detector. Each scan provides intensity data of the sample (2-D spatial-spatial) for a wavelength band and is repeated by tuning the filter so that the next wavelength band (1-D spectral) can be recorded.

In a spatiospectral-scanning imager, each scan provides intensity data of a diagonal slice of the datacube. This differs from a spatial- and spectral-scanning imager, which performs scanning in the orthogonal direction of the datacube. Relative motion between the detector and the sample is required to give a sequence of diagonal slices to build the datacube.²⁰

A snapshot imager acquires all the information required to build a datacube in one scan. Thus, no temporal scanning and relative motion between the sample and imager is required.²¹

The main advantage of a pushbroom imager over a spectral-scanning or snapshot imager is that it offers a much higher spectral resolution across a broad spectral range. However, its main drawback is the need to have relative motion between the sample and the detector, which can limit its data acquisition rate.

A few spatial-scanning HS imagers have been reported in literature for biomedical-related applications. Some of these imagers do not use a video camera (VC) in the system,^14,22 whereas others incorporate a VC in the setup for direct video imaging,^23,24 which has many benefits. Using a VC for direct video imaging gives a better visual representation with a color image, which can be used to verify the data after measurement. The VC and detector camera (DC) can be positioned such that both the cameras capture a focused image simultaneously. Using the VC, samples of different thicknesses can be easily positioned to maintain the same working distance. The VC allows the sample to be positioned precisely and this is especially important for a system with small field-of-view (FOV). Unwanted and repeated scanning can be prevented and this saves time and minimizes deterioration of the sample. However, having direct video imaging capability alone does not allow the user to pinpoint exactly which area in the FOV of the VC as the region of interest (ROI).

In this context, this paper reports the instrumentation, calibration, and the theoretical framework that used to set up a pushbroom HS imager incorporating a VC for both direct video imaging and user-selectable ROI. The advantages of using such a configuration include the benefits for direct video imaging mentioned earlier. The function introduced for user-selectable ROI allows the storing of the information from only within the ROI, minimizing measurement time, data size, and computational time. This precise mining of information from only within the ROI is accomplished by mechanical and digital means. While the top-to-bottom scanning of the ROI (height) is done by an automated scanning stage, the mining of data from only the spectral range of interest and within the width of the ROI is done by digital means.

2.

Instrumentation of the Pushbroom HS Imager

The proposed pushbroom HS imager’s design and configuration is shown in Fig. 1. The imager consists of a three-axis motorized stage (Physik Instrumente, $x$ and $y$-axes: M-112.2DG, $z$-axis: M-110.1DG) to position the sample. The $y$-axis stage is used to move the sample between each scan. Light from the sample then passes through the forelens (Navitar 2-50145 doublet lens) and is placed in a fine focus adapter (Navitar 2-16265). This adapter is attached to the bottom side of the quadrocular adapter (Nikon Y-QT), which houses a sliding mirror. The sliding mirror is initially pushed into the quadrocular adapter and directs light toward the VC (Path 1 in Fig. 1) before scanning commences. The VC (iDS UI-1550LE-C-HQ) has $1600\times 1200$ light sensitive pixels, allowing direct video imaging of the sample. The software developed allows the user to choose a particular region within this FOV as the ROI. After selection of the ROI, the sliding mirror is pulled out of the quadrocular adapter and light travels straight toward the spectrograph and the DC (Path 2 in Fig. 1). Scanning can now begin after the sliding mirror is pulled out. The spectrograph (Specim ImSpectors V10E, 400 to 1000 nm, 2.8-nm spectral resolution) is used for the dispersion of light and the DC (Andor EMCCD LucaEM DL-604M-OEM, 400-1000 nm), with $1004\times 1002$ light sensitive pixels, is used to record the spectral information.

Fig. 1

Schematic diagram of the proposed pushbroom hyperspectral imager.

3.

Instrumentation of the Pushbroom HS Imager

The calibration can be divided into three main parts (FOV, spectral, and position).

3.3.

Position Calibration

The VC and DC have different views of the sample. The VC has a rectangular view of the sample, whereas the DC has an LOV across the sample. The length of the DC’s LOV is also longer than the width of the VC’s FOV (Fig. 2). Thus, position calibration between these two cameras is required.

Fig. 2

Field-of-view (FOV) of video camera (VC) and line-of-view (LOV) of direct camera (DC).

3.3.1.

${\mathsf{Cal}}_{\mathsf{L}}$ and ${\mathsf{Cal}}_{\mathsf{R}}$

This calibration is done as the width across the sample viewed by VC is shorter than DC. ${\mathrm{Cal}}_{\mathrm{L}}$ and ${\mathrm{Cal}}_{\mathrm{R}}$ refer to the DC column indices (${\mathrm{DC}}_X$) corresponding to the extreme left and right views of the VC, respectively. The sample used is a United States Air Force (USAF) chart, placed such that the left edge of a dark square is along the extreme left view of the VC. By looking at the DC image, the dark square is easily identified. The ${\mathrm{DC}}_X$ which corresponds to the left side of the dark portion is ${\mathrm{Cal}}_{\mathrm{L}}$. This process is shown in Fig. 3. ${\mathrm{Cal}}_{\mathrm{L}}$ is found to be 224, which means that the left most view of the VC is imaged onto the 224th DC column index. ${\mathrm{Cal}}_{\mathrm{R}}$ is obtained using similar procedure and is found to be 777.

Fig. 3

${\mathrm{Cal}}_{\mathrm{L}}$ calibration.

3.3.2.

${\mathsf{Cal}}_{\mathsf{LOV}}$

This calibration is done to determine the VC row index (${\mathrm{VC}}_Y$), which shares the same view as the DC’s LOV. ${\mathrm{Cal}}_{\mathrm{LOV}}$ can be found by first looking at the DC view and then slowly changing the sample’s position until a change on the DC view is observed. This happens when the sample enters the DC’s LOV. ${\mathrm{Cal}}_{\mathrm{LOV}}$ is found to be 542. The DC has an LOV across the sample and this LOV corresponds to the 542th row from the top of the VC’s light sensitive pixel array.

4.

User Defined Parameters

These parameters give the user flexibility in using the system so that it can be faster and give only the required data for later analysis.

5.

Return Values/Vectors

All the steps and procedures mentioned in Secs. 3 and 4 are used to produce four return values and two vectors (to be described in detail in this section). They communicate with the DC and $y$-axis stage to collect data only from the user defined parameters.

5.1.

${\mathsf{X}}_{\mathsf{Min}}$ and ${\mathsf{X}}_{\mathsf{Max}}$

$X_{\mathrm{Min}}$ and $X_{\mathrm{Max}}$ refer to the ${\mathrm{DC}}_X$ which correspond to the left and right of the ROI, respectively. Each scan records data from the DC between $X_{\mathrm{Min}}$ and $X_{\mathrm{Max}}$ only. ${\mathrm{VC}}_X$ and ${\mathrm{DC}}_X$ are akin to different scales while referring to the same object (Fig. 4). Linear interpolation is used to determine the value of $X_{\mathrm{Min}}$ and $X_{\mathrm{Max}}$ in Eqs. (2) and (3), respectively. The ${\mathrm{Cal}}_{\mathrm{L}}$ and ${\mathrm{Cal}}_{\mathrm{R}}$ mentioned in Sec. 3.3.1 are used here. The DC does not recognize $X_{\mathrm{Max}}$. It requires the starting column index $X_{\mathrm{Min}}$ and the length of column $X_{\text{Length}}$, which is calculated using Eq. (4).

Eq. (2)

$$X_{\mathrm{Min}}=\mathrm{rd}[\frac{\text{Left}-1}{1600-1}({\mathrm{Cal}}_{\mathrm{R}}-{\mathrm{Cal}}_{\mathrm{L}})+{\mathrm{Cal}}_{\mathrm{L}}],$$

Eq. (3)

$$X_{\mathrm{Max}}=\mathrm{rd}[\frac{\text{Right}-1}{1600-1}({\mathrm{Cal}}_{\mathrm{R}}-{\mathrm{Cal}}_{\mathrm{L}})+{\mathrm{Cal}}_{\mathrm{L}}],$$

Eq. (4)

$$X_{\text{Length}}=X_{\mathrm{Max}}-X_{\mathrm{Min}}+1,$$

where rd means round off to nearest integer.

Fig. 4

$X_{\min }$ and $X_{\max }$.

6.

HyperSpec

A software based on LabView®, called HyperSpec, has been developed in-house. The control panel is shown in Fig. 5. It is used for the software interfacing of the VC, DC, and three-axis stage and incorporates all the points discussed in Secs. 3, 4, and 5. After calibration and entering the user-defined parameters, the scanning can begin. The return values are determined automatically, and the repeating process of stage movement and then DC data recording will also run on its own. After all the scanning has been completed, the stage places the sample back to the original position it was at just before scanning started.

Fig. 5

HyperSpec control panel.

7.

Data Rearrangement and Representation

The files saved are imported and processed by an in-house written script in MATLAB®. The script arranges the 2-D data to a single three-dimensional datacube. As data representation is more flexible and can vary depending on the needs, more parameters can be altered and customized. Many types of plots can be made available, such as spectrum plot, images at different wavelength bands and a datacube.

8.

Measurement and Results

The following measurements in this section were taken at maximum zoom where the full FOV of the VC is about $4.32\times 5.76{\mathrm{mm}}^2$ with a working distance of about 21.5 cm.

8.1.

VC for Selectable ROI

A USAF resolution chart is used in this section. A fiber-optic pigtailed source (Edmund Optics MI-150) was used for illumination. The full VC FOV before measurement and the selected ROI (Group 3 of the USAF chart) indicated by a black rectangle can be seen in Fig. 5. This section first shows the different plots of the results that can be acquired from each set of data (Figs. 6Fig. 7–8). This section also compares the image of the ROI selected to the spatial image captured by the system at a particular wavelength to validate whether or not the system is working well and capturing data only from the ROI.

Fig. 6

(a) Sequence of data acquisition and (b) datacube.

Fig. 7

(a) Cut datacube and (b) wavelength stack of bands 550:25:750 nm.

Fig. 8

Intensity mapping of nine selected spectral bands from the 756 total acquired bands ($100\times 677\text{pixels}$).

Figure 9 is made up of an image of the ROI, with two other identical images from the data at 650 nm placed beside and below it. The four dashed lines in this figure match features in the ROI to the same feature in the data. Therefore, it is observed that the system performed scanning only across the selected ROI, and only data in the ROI are saved. This validates the steps and formulas mentioned in Secs. 3 to 7. The longer vertical dotted line also shows that the ROI and data have the same orientation. Therefore, the DC, VC, and $y$-axis stage are all aligned with respect to each other. The VC is successfully implemented into the pushbroom HS imager for a user-selectable ROI to minimize data acquisition time and data size.

Fig. 9

Comparison of region of interest (ROI) and intensity mapping for validation of user-selectable ROI capability.

8.2.

Determining Lateral Resolution Using USAF Chart

This section uses the same set of data as in Sec. 8.1. From Fig. 10, the horizontal and vertical lines of Group 3 Element 5 can still be distinguished. Thus, the horizontal and vertical lateral resolution of the system at 650 nm in the basic configuration without any image enhancement is determined using Group 3 Element 5 of the USAF chart and is calculated to be $40\mu \mathrm{m}$.

Fig. 10

(a) ROI and (b) intensity mapping of 650 nm.

8.3.

Sample Analysis Demonstration Using Chicken Breast in Reflection Mode

Chicken breast tissue devoid of fat and skin is used, with a visible blood clot on the surface. This part of the chicken breast was chosen so that the blood clot can provide a contrast in the image. The sample on the glass slide and the ROI are shown in Figs. 11(a) and 11(b), respectively. The same illumination source as Sec. 8.1 was used (Edmund Optics MI-150) for conducting this study. Figure 12 shows the intensity mapping at four different wavelengths. The regions where 400 spectra were extracted and processed to represent the spectra of the blood clot and the chicken breast tissue are marked by the small white and black rectangles, respectively, in Figs. 11(b) and 12. Figure 13 shows the processed spectra of the chicken breast tissue and the blood clot and are found to be easily distinguishable from each other. These results indicate that such spectral data can be used as a data library to compare and identify unknown samples in the future.

Fig. 11

(a) Chicken breast tissue on glass slide and (b) ROI.

Fig. 12

Intensity mapping of (a) 550 nm, (b) 630 nm, (c) 670 nm, and (d) 850 nm.

Fig. 13

Spectra of blood clot and chicken breast tissue.

8.4.

Fluorescence Imaging

A Rhodamine 6G fluorescent film, which has been placed on a tissue phantom (Simulab Corporation), and the ROI are shown in Figs. 14(a) and 14(b), respectively. An excitation wavelength of 500 nm (NKT Photonics SuperK Extreme EXR-15, SuperK Select 4xVIS/IR) was used together with a beam expander unit so that the expanded beam covered the entire FOV of the VC. The measurement was taken with an exposure time of 150 ms and an EM gain of 10. The entire spectral range from 400 to 1000 nm was recorded, though Fig. 16 shows only a shorter spectral range for a better representation.

Fig. 14

(a) Rhodamine 6G fluorescent film on tissue phantom and (b) ROI.

The intensity mapping of 535, 563, and 585 nm is shown in Fig. 15 to illustrate the differences in fluorescence intensity at varying wavelengths. The fluorescence spectrum is calculated from the area within the black boxes in Figs. 14(b) and 15. Figure 16 shows the processed excitation and fluorescence spectra, each normalized with respect to itself. The orange solid line shows the fluorescence spectrum which is an average of 400 spectra from the region indicated by the black rectangle in Figs. 14(b) and 15. The green dotted line is the excitation spectrum measured separately from a piece of white paper.

Fig. 15

Intensity mapping of (a) 535 nm, (b) 563 nm (peak emission), and (c) 585 nm.

Fig. 16

Normalized excitation and fluorescence spectra.

The HSI of fluorescing samples is able to capture multiple fluorescent images at different wavelength bands. In this study, about 250 fluorescent images were captured between 500 to 700 nm (three of them shown in Fig. 15). Compared to the use of conventional imaging setup which uses a fluorescence filter to capture all the emission wavelengths in a single image, HSI provides much more information that can be used for a more accurate disease diagnosis. This can prove useful in disease diagnosis of the colon where the intensity and distribution of endogenous fluorophores are indicators of disease progression.²⁵

9.

Conclusion

A pushbroom HS imager which incorporates a VC not only for direct video imaging (benefits mentioned in Sec. 1) but also for a user-selectable ROI within the full imaging FOV of the VC is proposed and demonstrated in this paper. These concepts bring several benefits especially to a pushbroom HS imager. After selecting the ROI, scanning takes place only within the ROI. There is no unwanted scanning, thus minimizing data acquisition time as well as the data size. A smaller data size in turn translates to a shorter computational time in data processing and analysis. Similar applications can also be applied to spectral-scanning and snapshot imagers. However, it will not result in a shorter data acquisition time in spectral-scanning (number of scans depends on number of spectral band, not size of ROI) or a snapshot imager (only one scan required.). The use of VC for a user-selectable ROI presented in this paper tries to negate the pushbroom HS imager as being a relatively slower HS imager.

In the current configuration, the VC has an adjustable full imaging FOV using the fine focus adapter. The minimum and maximum full imaging FOV is about $4.32\times 5.76{\mathrm{mm}}^2$ (working distance of about 21.5 cm) and $5.17\times 6.89{\mathrm{mm}}^2$ (working distance of about 23.8 cm), respectively. The full FOV is also the maximum size of the ROI that can be selected by the user. The system has a maximum spectral range covering the visible to near-infrared light from 400 to 1000 nm and can detect 756 spectral bands within this spectral range. By using a DC and spectrograph suitable for imaging wavelengths more than 1000 nm, it is possible to extend the spectral range further into the infrared wavelengths. The horizontal and vertical lateral resolution of this system at maximum zoom without the use of any image enhancement is about $40\mu \mathrm{m}$. Such a lateral resolution makes the system suitable for use in biomedical imaging on tissue.

In reflection mode imaging, a common and relatively cheap quartz halogen white light source (Edmund Optics MI-150) was used. With respect to the maximum spectral range of interest (400 to 1000 nm), the bulb used in this light source has a poor transmittance from 400 to 500 nm and 800 to 1000 nm. Also within the same spectral region, the DC has a lower quantum efficiency. This can be seen in the spectral plot from the reflection mode (Fig. 13) where intensity counts below 450 nm and above 900 nm are always much lower compared to the central wavelengths. Without changing the DC, this issue can be resolved using a light source with a higher intensity in the extreme ends of the maximum spectral range of interest.

The experiments with the bio and fluorescent phantom samples shown here also demonstrate that this developed pushbroom HS imager that can be used as both reflection and fluorescence imaging modalities. The lateral resolution can be varied and improved using additional optical elements and by optodigital schemes. The use of this system can also be extended to other applications such as cellular scale biomedical imaging. In addition, by integrating the proposed configuration with a flexible probe scheme, it is expected to find potential endoscopic imaging applications as well.