1.

Introduction

Multispectral images present complementary information that typically includes visual-band (e.g., RGB or intensified) imagery and infrared imagery [e.g., near infrared (NIR) and long-wave infrared (LWIR)]. Imagine a night-time object assessment task that may be executed on the ground or by aircraft equipped with a multispectral imaging system. Multispectral images enable night vision (NV) but it is inconvenient to directly observe and analyze multiple images of a scene. Instead, analyzing the synthesized (fused or colorized) multisensory image is more informative and efficient for target recognition.¹ The fused multispectral imagery (in gray scale) can increase the reliability of interpretation^2,3 and supports machine analysis (computer vision), whereas the colorized multispectral imagery (in colors) improves observer situational awareness,⁴ reaction time,⁵ and perceptual analysis (human vision).⁶ This paper focuses on the discussion and comparisons of several NV coloring methods using multispectral images.

A NV colorization technique can produce colorized imagery with a naturalistic and stable color appearance by processing multispectral NV imagery. Although appropriately false-colored imagery is often helpful for human observers in improving their performance on scene classification and reaction time tasks,^6,7 inappropriate color mappings can also be detrimental to human performance.^5,8 A possible reason is lack of physical color constancy.⁵ Another drawback with false coloring is that observers need specific training with each of the false color schemes so that they can correctly and quickly recognize objects. With colorized night-time imagery rendered with natural colors, users should be able to readily recognize and identify objects without any training.

Toet⁹ proposed an NV colorization method that transfers the color characteristics of daylight imagery into multispectral NV images. Essentially, this color-mapping method matches the statistical properties (i.e., mean and standard deviation) of the NV imagery to that of a natural daylight color image (manually selected as the “target” color distribution). Thus, this method is referred as to “statistic matching.” However, this color-mapping method colorizes the image regardless of scene content, and thus the accuracy of the coloring is dependent on how well the target and source images are matched. In other words, the statistic-matching method weighs the local regions of the source image by the “global” color statistics of the target image, and thus yields less naturalistic results (e.g., biased colors) for images containing regions that differ significantly in their colored content.

To address this bias problem in global coloring, Zheng and Essock¹⁰ presented a “local coloring” method that can colorize the NV images to resemble daylight imagery. The local-coloring method renders the multispectral images with natural colors segment by segment (i.e., “segmentation-based”), and also provides an automatic association between the source and target images (i.e., by avoiding the manual scene-matching in global coloring).

The segmentation-based colorization can usually produce a vivid NV image closely resembling the colors in a natural scene. However, the segmentation-based coloring procedure involves many processes and heavy computations, such as image segmentation and pattern classification. Zheng¹¹ recently introduced a channel-based color-fusion method, which is fast enough for real-time applications. Note that the term “color fusion” in this paper refers to combing multispectral images into a color-version image with the purpose of resembling natural scenes.

Hogervorst and Toet^12,13 recently proposed a new color-mapping method using a look-up table (LUT). The LUT is created between a false-colored image (formed with multispectral NV images) and its color reference image (aiming at the same scene but taken during the daytime). The colors in the resulting colored NV image resemble the colors in the daytime color image. This LUT-mapping method, which runs fast for real-time implementations, is summarized along with the statistic-matching method in their recent paper.¹³

The quality of colorized images can be assessed by subjective/objective measures. However, subjective evaluation normally costs time and resources and should be related to a standard, such as the National Imagery Interpretability Rating Scale (NIIRS). Moreover, the subjective evaluation methods cannot be readily and routinely used for real-time and automated systems. On the other hand, objective evaluation metrics can automatically and quantitatively measure the image qualities.¹⁴ In the past decade, many objective metrics for grayscale image evaluations have been proposed.^15–17 However, the metrics for grayscale images cannot be directly extended to the evaluations of colorized images. Recently, some objective evaluations of color images have been reported in the literature. To objectively assess a color-fusion method, Tsagaris¹⁸ proposed a color image fusion measure (CIFM) by using the amount of common information between the source images and the colorized image, and also the distribution of color information. Yuan et al.¹⁹ presented an objective evaluation method for visible and infrared color fusion utilizing four metrics: image sharpness metric, image contrast metric, color colorfulness metric, and color naturalness metric. In this paper, we introduce an objective evaluation index (OEI) to quantitatively evaluate the colorized images. Given a reference (daylight color) image and several versions of the colorized NV images from different coloring techniques, all color images are first converted into International Commission on Illumination (CIE) LAB space, with dimension $\mathit{L}$ for lightness and $\mathit{a}$ and $\mathit{b}$ for the color-opponent dimensions.²⁰ Then the OEI metric is computed with the four established metrics, phase congruency metric (PCM), gradient magnitude metric (GMM), image contrast metric (ICM), and color natural metric (CNM).

Certainly, a color presentation of multispectral NV images can provide a better visual input for human users. Users expect the colored images to closely resemble natural daylight pictures; along with a coloring process fast enough for real-time applications. In this paper, six NV coloring methods (i.e., color fusion, statistic matching, histogram matching, joint histogram matching, LUT-mapping, segmentation-based coloring) are explored and compared, using both qualitative and quantitative evaluations and employing a new color-mapping method of joint-histogram matching that is developed in the paper. Conclusions are drawn based the experimental results. The rest of paper is organized as follows. Multispectral image preprocessing and color space transform are briefly described in Sec. 2, channel-based color fusion is reviewed in Sec. 3, color-mapping methods are presented in Sec. 4, quantitative evaluation metrics are defined in Sec. 5 and the experiments and discussions are given in Sec. 6. Conclusions are drawn in Sec. 7.

2.

Preprocessing and Color Space Transform

All NV colorization methods require preprocessing and color space transform, which are briefly reviewed in this section.

3.

Color Fusion Methods

A fast color fusion method, termed as channel-based color fusion, was introduced to facilitate real-time applications.¹¹ The term of “color fusion” means combing multispectral images into a color-version image with the purpose of resembling natural scenes. Relative to the “segmentation-based colorization” (refer to Sec. 4.5), color fusion performs a trade-off between color realism with processing speed.

The general framework of channel-based color fusion is as follows: prepare for color fusion, preprocessing (denoising, normalization, and enhancement) and image registration; form a color fusion image by properly assigning multispectral images to red, green, and blue channels; then fuse multispectral images (gray fusion) using $a\mathrm{DWT}$ algorithm;²⁴ and replace the value component of color fusion in color natural metric (HSV) color space with the gray-fusion image, and finally transform the fused image back to RGB space.

In NV imaging, there may be several bands of images available, for example, visible (RGB), image intensified (II), NIR, medium-wave infrared (MWIR), and LWIR. On the basis of the available images and the context, we only discuss two-band color fusions: ($\mathrm{II}\oplus \mathrm{LWIR}$), and ($\mathrm{NIR}\oplus \mathrm{LWIR}$), with the symbol ‘$\oplus$’ denotes the fusion of multiband images.

4.

Color-Mapping Methods

All color-mapping methods described in this section are performed in $l\alpha \beta$ color space. Thus, the color space conversion from RGB to $l\alpha \beta$ must be done prior to color mapping, and then the inverse transform to RGB space is necessary after the mapping. Certainly, two images, a source image and a target image, are involved in a color-mapping process (refer to Fig. 1). The source image is usually a color-fusion image (in Secs. 4.1 to 4.3) or a false-colored image (in Sec. 4.4); while the target image is normally a daylight picture containing the similar scene. The target image may have a different resolution as depicted in Secs. 4.1 to 4.3; however, the LUT described in Sec. 4.4 is established using the registered target (reference) image.

Fig. 1

Diagram of joint histogram matching [demonstrated with joint-HM($\beta \alpha$)].

4.1.

Statistic Matching

A statistic matching (stat-match) is used to transfer the color characteristics from natural daylight imagery to false color night-vision imagery, which is formulated as:

Eq. (8)

$${\mathit{I}}_C^k=({\mathit{I}}_S^k-{\mathit{\mu }}_S^k)\frac{{\mathit{\sigma }}_T^k}{{\mathit{\sigma }}_S^k}+{\mathit{\mu }}_T^k,\mathrm{for}k=\{l,\alpha ,\beta \},$$

where ${\mathit{I}}_C$ is the colored image, ${\mathit{I}}_S$ is the source (false-color) image in $l\alpha \beta$ space; $\mathit{\mu }$ denotes the mean and $\sigma$ denotes the standard deviation; the subscripts ‘S’ and ‘T’ refer to the source and target images, respectively; and the superscript ‘$k$’ is one of the color components: $\{l,\alpha ,\beta \}$.

After this transformation, the pixels comprising the multispectral source image have means and standard deviations that conform to the target daylight color picture in $l\alpha \beta$ space. The colored image is transformed back to the RGB space [refer to Fig. 2(e)] through the inverse transforms [i.e., $l\alpha \beta$ space to the $\mathit{LMS}$, exponential transform from $\mathit{LMS}$ to LMS, and LMS to RGB, refer to Eqs. (2) to (5].⁹

Fig. 2

Illustration of color-mapping techniques using only daylight RGB pictures: (a) and (b) source (Lotus) and target (Tulip) images; (c) and (d) the cumulative histograms of Luminance ($l$), and the 1-D histograms from the 2-D JHs of Beta-Alpha ($\beta \alpha$) in processing (g); (e) and (f) the mapped images using statistic-matching, and histogram-matching (HM), respectively; (g)–(i) the mapped images using joint-HM ($\beta \alpha$), statistic-matching then joint-HM ($\beta \alpha$), and joint-HM ($l\alpha$), respectively. Notice that the contrasts of all color images were increased by 10%.

4.4.

Look-Up Table

Hogervorst and Toet^12,13 proposed a color-mapping method using a look-up table (LUT). The LUT is created using one image pair: a false-colored image (formed with two-band NV images) and a reference (i.e., target) daylight image. This method yields a colored NV image similar to the daytime image colors. The implementation of this LUT method is described as follows.

• (1) Create a false-colored image (of three color planes) by assigning LWIR to R, NIR image to G plane, and zeros to B, respectively;

• (2) Build RG color map (i.e., a $256\times 256\mathrm{LUT}$) and convert the false-colored image to an indexed image (0 to 65535) associated with the RG color map;

• (3) For all pixels in the indexed false-colored image whose index value equals 0:

  • (a) Locate all corresponding pixels in the reference (i.e., target) color image (that must be strictly aligned with the false-colored image);

  • (b) Calculate the averaged $l\alpha \beta$ values of those corresponding pixels and then convert them back to RGB values;

  • (c) Assign the RGB values to index 0 in the look-up table;

• (4) Vary the index value from 1 to 65535 and repeat the processes described in Step 3. At the end, the LUT will be established.

Once the LUT is created, the LUT-based mapping procedure is simple and fast [see Fig. 3(i)], and thus can be deployed in real-time. However, the LUT creation thoroughly relies on the aligned reference image corresponding to the same scene. Any misalignment, using a different reference color image, or coloring a different NV imagery (i.e., aiming at a different direction), will usually result a degraded colorization. To make the LUT colorization independent of viewing direction, Hogervorst and Toet^12,13 further suggested deriving the LUT table from more than one corresponding image pair (false-colored versus daylight) such that all materials relevant for a given surroundings are represented in the imagery from which the LUT is derived.

Fig. 3

Night-vision coloring comparison (Case# ST014 in NV-set 1—taken at sunset; $640\times 480\mathrm{pixels}$): (a-c) Color RGB, NIR, and LWIR images, respectively; (d)–(f) the colorized images using channel-based color fusion of ($\mathrm{NIR}\oplus \mathrm{LWIR}$), statistic-matching, and histogram-matching, respectively; (g)–(i) the colorized images using joint-HM, statistic-matching then joint-HM, and LUT-mapping [$\mathrm{reference}=(a)$], respectively. The settings in the color-mappings of (e)–(h) are $\mathrm{source}=(d)$ and $\mathrm{target}=(a)$. Notice that the contrasts of all color images were increased by 10%.

5.

Quantitative Metrics for Colorization Evaluations

Three image quality metrics for grayscale images and one metric for color images are reviewed in Sec. 5.1. A new objective metric, termed object evaluation index²⁸ (OEI), is introduced in Sec. 5.2, which is defined with the four metrics. The color-related metrics are defined in the CIELAB space, where CIE stands for the International Commission on Illumination and LAB is for $L^{*}{a}^{*}{b}^{*}$. The perceptually uniform CIELAB space consists of an achromatic luminosity component $L^{*}$ (black-white) and two chromatic values $a^{*}$ (green-magenta) and $b^{*}$ (blue-yellow). The coordinates $L^{*}{a}^{*}{b}^{*}$ (CIE 1976) can be calculated using the CIE $XYZ$ tri-stimulus values.²⁰

5.1.

Four Image Quality Metrics

5.1.1.

Phase congruency metric

The phase congruency (PC) model is also called the “local energy model” developed by Morrone et al.²⁹ This model postulates that the features in an image are perceived at the points where the Fourier components are maximal in phase. Based on the physiological and psychophysical evidences, the PC theory provides a simple but biologically plausible model of how mammalian visual systems detect and identify the features in an image.^29–32 The PC can be considered as a significance measure of local structures in an image.

According to the definition of PC,²⁹ there are many different implementations developed so far. A widely-used method developed by Kovesi³⁰ is adopted in this paper. Given a 1-D image $f(x)$, $M_n^e$ and $M_n^o$ represent the even-symmetric and odd-symmetric filters at scale $n$, respectively. $M_n^e$ and $M_n^o$ form a quadrature pair: $e_n(x)$ and $o_n(x)$. Responses of the quadrature pair form a response vector:

Eq. (11a)

$$\left[\begin{array}{c}{e}_n(x)\\ o_n(x)\end{array}\right]=\left[\begin{array}{c}f(x)*M_n^e\\ f(x)*M_n^o\end{array}\right],$$

and the local amplitude at scale $n$ is

Eq. (11b)

$$A_n(x)=\sqrt{e_n^2(x)+o_n^2(x)}.$$

Let

Eq. (11c)

$$F(x)=\sum _n{e}_n(x),H(x)=\sum _n{o}_n(x).$$

The 1-D PCM can be computed as

Eq. (11-d)

$$\mathrm{PC}(x)=\frac{\sqrt{F^2(x)+H^2(x)}}{{\sum }_n{A}_n(x)+\epsilon },$$

where $\epsilon$ is a small positive constant.

In order to calculate the quadrature pair of filters $M_n^e$ and $M_n^o$, Gabor filters³³ or log-Gabor filters³⁴ can be applied. In this paper, we use log-Gabor filters (e.g., wavelets at scale $n=4$) due to its following two features: log-Gabor filters, by definition, have no direct current (DC) component; and the transfer function of the log-Gabor filter has an extended tail at the high frequency end, which makes it more capable to encode natural images than ordinary Gabor filters.³⁵ The transfer function of a log-Gabor filter in the frequency domain is

Eq. (12a)

$$G(\omega )=e^{-\frac{{[\text{log}(\omega /{\omega }_0)]2,}^{}}{2{\sigma }_r^2}},$$

where ${\omega }_0$ is the filter’s center frequency and ${\sigma }_r$ controls the filter’s bandwidth.

To compute the PCM of 2-D grayscale images, we can apply the 1-D analysis over several orientations and then combine the results according to some rules that optimize performance. The 1-D log-Gabor filters described above can be extended to 2-D ones by applying a Gaussian function across the filter perpendicular to its orientation.^30,34,36,37 The 2-D log-Gabor function has the following transfer function

Eq. (12b)

$$G_2(\omega ,{\theta }_j)=e^{-\frac{{[\text{log}(\omega /{\omega }_0)]}^2}{2{\sigma }_r^2}}·e^{-\frac{{(\theta -{\theta }_j)}^2}{2{\sigma }_{\theta }^2}},$$

where ${\theta }_j=\frac{j\pi }{2J}$ and $j=0,1,2,\dots ,J-1$ is the number of orientations and ${\sigma }_{\theta }$ determines the filter’s angular bandwidth. By modulating ${\omega }_0$ and ${\theta }_j$ and convolving $G_2$ with the 2-D image, we get a set of responses at each point $(x,y)$ as $[e_{n,{\theta }_j}(x,y),o_{n,{\theta }_j}(x,y)]$. The local amplitude at scale $n$ and orientation ${\theta }_j$ is

Eq. (13a)

$$A_{n,{\theta }_j}=\sqrt{e_{n,{\theta }_j}^2(x,y)+o_{n,{\theta }_j}^2(x,y)}$$

and the local energy along orientation ${\theta }_j$ is

Eq. (13b)

$$E_{{\theta }_j}=\sqrt{F_{{\theta }_j}^2(x,y)+H_{{\theta }_j}^2(x,y)},$$

where

Eq. (13c)

$$F_{{\theta }_j}(x,y)=\sum _n{e}_{n,{\theta }_j}(x,y),H_{{\theta }_j}(x,y)=\sum _n{o}_{n,{\theta }_j}(x,y).$$

The 2-D PCM at $(x,y)$ is defined as

Eq. (13d)

$${\mathrm{PC}}_{2D}(x,y)=\frac{\sum _j{E}_{{\theta }_j}(x,y)}{{\sum }_n{\sum }_j{A}_{n,{\theta }_j}(x,y)+\epsilon },$$

where $\epsilon$ is a small positive constant. It should be noted that ${\mathrm{PC}}_{2D}(x,y)$ is a real number within [0,1]. The PCM of an image is defined as

Eq. (13e)

$$\text{PCM}=\frac{1}{MN}\sum _{(x,y)}{\mathrm{PC}}_{2D}(x,y)=\frac{1}{MN}\sum _{(x,y)}\frac{\sum _j{E}_{{\theta }_j}(x,y)}{{\sum }_n{\sum }_j{A}_{n,{\theta }_j}(x,y)+\epsilon },$$

where $M\times N$ is the size of the image. The range of PCM is [0,1].

5.1.2.

Gradient magnitude metric

The image gradient magnitude (GM) is computed to encode contrast information. So, PC and GM are complementary and reflect different aspects of the human visual system in assessing the local image quality, with GM measuring the sharpness of an image. The perception of sharpness is related to an image’s clarity of detail. Image gradient computation is a traditional topic in image processing and gradient operators can be expressed by convolution masks. One commonly used gradient operator is the Sobel operator. The partial derivatives of image $f(x,y)$, $G_x$ and $G_y$, along horizontal and vertical directions using the Sobel operators are

Eq. (14a)

$$G_x=\frac{1}{4}\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right]*f(x,y),G_y=\frac{1}{4}\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right]*f(x,y).$$

The GM of $f(x,y)$ at pixel $(x,y)$ is defined as

Eq. (14b)

$$G(x,y)=\sqrt{G_x^2+G_y^2}.$$

The averaged GM over all pixels is called image GMM,

Eq. (14c)

$$\text{GMM}=\frac{1}{MN}\sum _{x,y}G(x,y)=\frac{1}{MN}\sum _{x,y}\sqrt{G_x^2+G_y^2},$$

where $M\times N$ is the size of the image.

5.1.3.

Image contrast metric

An image with excellent contrast has a wide dynamic range of intensity level and appropriate intensity. Both the dynamic range of intensity level or the overall intensity distribution of the image can be provided by a histogram. A global contrast metric¹⁹ is proposed using the histogram character. The histogram of image with levels in the range [0, $N\text{\hspace{0.17em}−\hspace{0.17em}1}$] is a frequency-distribution function defined as the overall intensity distribution of an image

Eq. (15a)

$$h(X_k)=n_k,$$

where $X_k$ is the $k$’th level of input and $n_k$ is the number of the pixels in the image having level $X_k$. The probability density function (PDF) is computed by

Eq. (15b)

$$P(X_k)=n_k/n,$$

where $n$ is the total number of the pixels of the image. The dynamic range value $\beta$ is defined as

Eq. (15c)

$$\beta =\sum _{k=0}^{L-1}S(X_k),$$

where

Eq. (15d)

$$S(X_k)=\{\begin{array}{ll}1,& \mathrm{if}\hspace{0.17em}P(X_k)>0\\ 0,& \text{otherw}\text{ise}\end{array}.$$

The dynamic range matrix $\alpha$ of histogram is defined as

Eq. (15e)

$$\alpha =\frac{\beta }{2N-\beta },$$

where $\alpha \in [0,1]$. Note that a larger value of $\alpha$ means a wider dynamic range in the histogram, which leads to better contrast. The local image contrast metric is defined as

Eq. (15f)

$$C=\alpha \sum _{k=0}^{N-1}\frac{X_k}{N}P(X_k).$$

For color images, the image contrast metric is determined by both gray contrast and color contrast. Because human perception is more sensitive to the luminance on contrast evaluation, we employ the $L^{*}$ channel in the CIELAB space to evaluate the color contrast. Thus, image contrast is determined by the histogram of gray intensity and the histogram of color luminance $L^{*}$ (see Fig. 4). For the gray intensity $I$, the gray contrast metric is defined as

Eq. (16a)

$$C_g={\alpha }_I\sum _{k=0}^{N_I-1}\frac{I_k}{N}P(I_k),$$

where ${\alpha }_I$ and $P(I_k)$ can be calculated as above for gray intensity. For the $L^{*}$ channel, the color contrast metric is

Eq. (16b)

$$C_c={\alpha }_c\sum _{k=0}^{N_{L^{*}}-1}\frac{L_k^{*}}{N_{L^{*}}}P(L_k^{*}),$$

where ${\alpha }_c$ and $P(L_k^{*})$ can be calculated as above for the $L^{*}$ channel. The global image contrast metric (ICM) is defined as

Eq. (16c)

$$\text{ICM}=\sqrt{{\omega }_1{C}_g^2+{\omega }_2{C}_c^2},$$

where ${\omega }_1$ and ${\omega }_2$ are the weights of $C_g$ and $C_c$. For simplicity, we choose ${\omega }_1={\omega }_2=$ 0.5. ICM varies within [0,1]. The evaluation of image contrast metric of color fusion image is shown in Fig. 4.

Fig. 4

Diagram of calculation of the contrast metric.

5.1.4.

Color natural metric

Given a daylight image $f_1(x,y)$ and a colorized image $f_2(x,y)$, if a colorized image is similar to the daylight image then the colorized image is considered good quality. Since a human is sensitive to hue in addition to luminance, we compare the $a^{*}$ and $b^{*}$ channels of the reference image with that of the colorized image using the gray relational analysis (GRA) theory.³⁸

We first convert two images, $f_1$ and $f_2$, to $L^{*}{a}^{*}{b}^{*}$ space. $L_i^{*}(x,y)$, $a_i^{*}(x,y)$, and $b_i^{*}(x,y)$ are the $L^{*}{a}^{*}{b}^{*}$ values of $f_i$ at pixel $(x,y)$. The gray relation coefficient between $a_1^{*}$ and $a_2^{*}$ at pixel $(x,y)$ is defined as

Eq. (17a)

$${\xi }_a(x,y)=\frac{\underset{i}{\text{min}}\underset{j}{\text{min}}|a_1^{*}(i,j)-a_2^{*}(i,j)|+0.5\underset{i}{\text{max}}\underset{j}{\text{max}}|a_1^{*}(i,j)-a_2^{*}(i,j)|}{|a_1^{*}(x,y)-a_2^{*}(x,y)|+0.5\underset{i}{\text{max}}\underset{j}{\text{max}}|a_1^{*}(i,j)-a_2^{*}(i,j)|+\epsilon },$$

where $\epsilon$ is a small positive constant. The gray relation coefficient between $b_1^{*}$ and $b_2^{*}$ at pixel $(x,y)$ is defined as

Eq. (17b)

$${\xi }_b(x,y)=\frac{\underset{i}{\text{min}}\underset{j}{\text{min}}|b_1^{*}(i,j)-b_2^{*}(i,j)|+0.5\underset{i}{\text{max}}\underset{j}{\text{max}}|b_1^{*}(i,j)-b_2^{*}(i,j)|}{|b_1^{*}(x,y)-b_2^{*}(x,y)|+0.5\underset{i}{\text{max}}\underset{j}{\text{max}}|b_1^{*}(i,j)-b_2^{*}(i,j)|+\epsilon }.$$

In the definitions of ${\xi }_a(x,y)$ and ${\xi }_b(x,y)$, $\min (\cdot )$ and $\max (\cdot )$ are operated over whole image. However, it is possible that $\min (\cdot )$ and $\max (\cdot )$ are operated over a small neighborhood of $(x,y)$. The gray rational degrees of $a^{*}$ and $b^{*}$ information for two images are defined as

Eq. (17c)

$$R_a=\sum _{(x,y)}\omega (x,y){\xi }_a(x,y),$$

Eq. (17d)

$$R_b=\sum _{(x,y)}\omega (x,y){\xi }_b(x,y),$$

where $\omega (x,y)$ is the weight of the gray rational coefficient, which satisfies

Eq. (17e)

$$\sum _{(x,y)}\omega (x,y)=1.$$

For simplicity, we choose $\omega (x,y)=\frac{1}{M\times N}$ where $M$ and $N$ are the length of vectors $x$ and $y$, respectively. The CNM is defined as

Eq. (17f)

$$\text{CNM}=\sqrt{R_a{R}_b}.$$

CNM varies within [0,1]; the larger the CNM, the more similar the two images.

5.2.

Objective Evaluation Index

With the four metrics defined in Sec. 5.1, a new OEI is proposed to quantitatively evaluate the qualities of colorized images. Given the reference image $f_1$ and the colorized image $f_2$, the OEI is calcualted in two steps. First the local similarity maps of the two images are computed, and then the similarity maps are integrated into a single similarity score.

The two images are first converted into $L^{*}{a}^{*}{b}^{*}$ space. For $L^{*}$ information, the PC maps are calculated and denoted as ${\mathrm{PC}}_1$ and ${\mathrm{PC}}_2$ for $f_1$ and $f_2$ images, respectively. The similarity measure, $S_{PC}$, between ${\mathrm{PC}}_1$ and ${\mathrm{PC}}_2$ at pixel $(x,y)$ is defined as

Eq. (18a)

$$S_{\mathrm{PC}}(x,y)=\frac{2{\mathrm{PC}}_1(x,y){\mathrm{PC}}_2(x,y)+K_1}{{\mathrm{PC}}_1^2(x,y)+{\mathrm{PC}}_2^2(x,y)+K_1},$$

where $K_1$ is a positive constant. In practice, the determination of $K_1$ depends on the dynamic range of PC values. $S_{PC}$ varies within [0,1]. Similarly, the similarity measure based on the two GM values is defined as

Eq. (18b)

$$S_G(x,y)=\frac{2G_1(x,y)G_2(x,y)+K_2}{G_1^2(x,y)+G_2^2(x,y)+K_2},$$

where $K_2$ is a positive constant. $S_G$ varies within [0,1]. Then, $S_{\mathrm{PC}}(x,y)$ and $S_G(x,y)$ are combined into one similarity measure, $S_L(x)$, where subscripts $L$ is for $L*a*b$ space, as follows

Eq. (18c)

$$S_L(x,y)={[S_{\mathrm{PC}}(x,y)]{\lambda }_1}^{}{[S_G(x,y)]{\lambda }_2}^{},$$

where ${\lambda }_1$ and ${\lambda }_2$ are parameters to adjust the relative importance of PC and GM features.

With the aid of the similarity $S_L(x,y)$ at each pixel $(x,y)$, the overall similarity between $f_1$ and $f_2$ can be calculated with the averaged $S_L(x,y)$ over all pixels. However, the image saliency (i.e., local significance) usually varies with the pixel location. For example, edges convey more crucial information than smooth areas. Specifically, a human is sensitive to phase congruent structures,³⁹ and thus the larger $\mathrm{PC}(x,y)$ value between $f_1$ and $f_2$ implies a higher impact on evaluating the similarity between $f_1$ and $f_2$ at location $(x,y)$. Therefore, we use ${\mathrm{PC}}_{\max }(x,y)=\text{max}[{\mathrm{PC}}_1(x,y),{\mathrm{PC}}_2(x,y)]$ to weigh the importance of $S_L(x,y)$ in formulating the overall similarity. Accordingly, the OEI between $f_1$ and $f_2$ is defined as follows

Eq. (19a)

$$\text{OEI}={\left(\frac{\sum _{(x,y)}{\mathrm{PC}}_{\max }(x,y)S_L(x,y)}{\sum _{(x,y)}{\mathrm{PC}}_{\max }(x,y)}\right)}^{{\gamma }_1}\times {(S_{\text{ICM}})}^{{\gamma }_2}\times {(\text{CNM})}^{{\gamma }_3},$$

where

Eq. (19b)

$${\mathrm{PC}}_{\max }(x,y)=\text{max}[{\mathrm{PC}}_1(x,y),{\mathrm{PC}}_2(x,y)],$$

Eq. (19c)

$$S_{\text{ICM}}=\frac{2\mathrm{IC}M(f_1)\times \text{ICM}(f_2)+K_3}{\text{ICM}{(f_1)2}^{}+\text{ICM}{(f_2)2}^{}+K_3},$$

CNM is previously defined in Eq. (17f) and $K_3$ and ${\gamma }_i$ ($i=1$, 2, 3) are positive constants. The diagram of calculating OEI is shown in Fig. 5. The range of OEI is [0,1]. The larger the OEI value of a colorized image is, the more similar (i.e., the better) the colorized image is to the reference image. Error pooling is the integration of methods with tradeoffs between ${\gamma }_1$, ${\gamma }_2$, and ${\gamma }_3$.

Fig. 5

Diagram of calculating OEI in $L^{*}{a}^{*}{b}^{*}$ space.

${\gamma }_1$, ${\gamma }_2$, and ${\gamma }_3$ are the weights of three components in the OEI metric, with the selection of ${\gamma }_i$ being critical for the OEI calculation. The values of ${\gamma }_i$ are empirically decided, and the typical values of ${\gamma }_1$ and ${\gamma }_2$ are between $0.8\sim 1.1$ and ${\gamma }_3$ is between $0.05\sim 0.2$. $K_i$ ($i=1$, 2, 3) are constants to increase the metric stability. In our experiments presented in Sec. 6, we chose ${\gamma }_1={\gamma }_2=1$, ${\gamma }_3=0.2$; $K_1=0.85$, $K_2=160$, $K_3=0.001$; and ${\lambda }_1={\lambda }_2=1$.

6.

Experimental Results and Discussions

To demonstrate and compare different color-mapping methods, two daylight pictures were used as source [Fig. 2(a); Lotus] and target [Fig. 2(b); Tulip] images, respectively (collected by authors). The colored results using statistic matching (stat-match), histogram matching (HM), joint histogram matching [joint-$\mathrm{HM}(\beta \alpha )$], and stat-match then joint-HM ($\beta \alpha$) are presented in Fig. 2(e) to 2(h), respectively. Figure 2(e) shows the background (water, leaf) painted with the Tulip’s colors; whereas Fig. 2(f) appears oversaturated in colors (i.e., color distortion). The Lotus shown in Fig. 2(g) [or Fig. 2(h)] has the closest colors to the Tulip but its background colors are altered. Two histograms of “Mapped” and “Target” shown Fig. 2(d) indicate the color difference between the mapped image and the target image. This result may imply that a source image can be ideally colorized when its color distribution (e.g., histogram) is similar with that of a target image. Another result of the joint-$\mathrm{HM}(l\alpha )$ is exhibited in Fig. 2(i), where Lotus shows in light and pure colors but its background (water) is distorted. Notice that the following experimental results of joint histogram matching were conducted by using joint-$\mathrm{HM}(\beta \alpha )$.

Two sets of multispectral NV images were used in our experiments, which were taken at night-time and referred as to “NV-set 1” and “NV-set 2,” respectively. In NV-set 1, three triplets of multispectral images (as shown Figs. 3, 6, and 7 collected at Alcorn State University), color RGB, NIR, and LWIR were colored by using different coloring methods as described in Sec. 4. The three-band input images are shown in Figs. 3, 6, and 7(a) to 7(c), respectively with the image resolutions given in the figure captions. The RGB images and LWIR images were taken by a FLIR SC620 two-in-one camera, which has a LWIR camera (of $640\times 480\mathrm{pixel}$ original resolution and 7.5 to 13 μm spectral range) and an integrated visible-band digital camera ($2048\times 1536\mathrm{pixel}$ original resolution). The NIR images were taken by a FLIR SC6000 camera ($640\times 512\mathrm{pixel}$ original resolution and 0.9 to 1.7 μm spectral range). Two cameras (SC620 and SC6000) were placed on the same fixture and turned to aim at the same location. The images were captured during sunset and dusk during autumn.

Fig. 6

Night-vision coloring comparison (Case# AT008 in NV-set 1—taken at sunset; $640\times 480\mathrm{pixels}$): (a–c) Color RGB, NIR, and LWIR images, respectively; (d)–(f) the colorized images using channel-based color fusion of ($\mathrm{NIR}\oplus \mathrm{LWIR}$), statistic-matching, and histogram-matching, respectively; (g)–(i) the colorized images using joint-HM, statistic-matching then joint-HM, and LUT-mapping [$\mathrm{reference}=(a)$], respectively. The settings in the color-mappings of (e)–(h) are $\mathrm{source}=(d)$ and $\mathrm{target}=(a)$. Notice that the contrasts of all color images were increased by 10%, and the brightness of (a) and (i) were increased by 10%.

Fig. 7

Night-vision coloring comparison (Case# AT012 in NV-set 1—taken at dusk; $640\times 480\mathrm{pixels}$): (a–c) Color RGB, NIR, and LWIR images, respectively; (d)–(f) the colorized images using channel-based color fusion of ($\mathrm{NIR}\oplus \mathrm{LWIR}$), statistic-matching, and histogram-matching, respectively; (g)–(i) the colorized images using joint-HM, statistic-matching then joint-HM, and joint-HM with different settings [$\mathrm{source}=(d)$, $\mathrm{target}=$ Fig. 6(a)], respectively. The settings in the color-mappings of (e)–(h) are $\mathrm{source}=(d)$ and $\mathrm{target}=$ Fig. 3(a). Notice that the contrasts of all color images were increased by 10%, and the brightness of (a) were increased by 20%.

Of course, image registration and fusion as described in Sec. 2.1 were applied to the three band images shown in Figs. 3, 6, and 7, where manual alignment was employed to the RGB image shown in Fig. 7(a) since it is so dark and noisy. To better present the color images (including the daylight RGB images and the colorized NV images), contrast and brightness adjustments (as described in figure captions) were applied. Notice that piece-wise contrast stretching [Eq. (1)] was used for NIR enhancement. As referred in Eq. 7(d), the fused images (shown elsewhere⁹) were obtained using the $a\mathrm{DWT}$ algorithm.²⁴ The channel-based color fusion [defined in Eqs. (7)] was applied to the NIR and LWIR images [shown in Figs. 3, 6, and 7(b) to 7(c)], and the results are illustrated in Figs. 3, 6, and 7(d). The resulted images from two-band color fusion [Figs. 3, 6, and 7(d)] resemble natural colors, which makes scene classification easier. The paved ground appears reddish since it has strong heat radiations (at dusk) and thus causes strong responses in LWIR images. In the color-fusion images, the trees, buildings and grass can be easily distinguished from ground (parking lots) and sky. The car is clearly identified in Fig. 7(d), where the water area (between ground and trees and shown in cyan color) is certainly noticeable. However, it is hard to realize any water area in the original images [Fig. 7(a) to 7(c)].

All color-mapping methods were applied to the three triplets and their results are presented in Figs. 3, 6, and 7. The source images are the color-fusion images [Figs. 3, 6, and 7(d)], while the target images are the color RGB images [Figs. 3 and 6(a)]. Figure 7(a) cannot be used as a target image as it is too dark and noisy. Figures 3, 6, and 7(e) show the colored images with the statistic-matching method, which are more similar to the daylight pictures in contrast with the color-fusion images. The three results [Figs. 3, 6, and 7(e)] are equivalently good, which means that the statistic matching is reliable. The histogram matching results shown in Figs. 3, 6, and 7(f) are oversaturated, which turns to be more suitable for segmentation-based colorization [see Fig. 8(c) and 8(g)]. The joint histogram matching [i.e., joint-$\mathrm{HM}(\beta \alpha )$] is illustrated in Figs. 3, 6, and 7(g), where the mapped images are better than the color fusions, but preserve the reddish colors which existed in the source images. Figure 7(i) is also a colored image using joint-$\mathrm{HM}(\beta \alpha )$ by choosing [$\mathrm{target}=$ Fig. 6(a)], which appears slightly better than Fig. 7(g) [$\mathrm{target}=$ Fig. 3(a)]. The comparable results [shown in Fig. 7(g) and 7(i)] demonstrate that the color-mapping methods can flexibly choose a target image with similar scenery. The “stat-match then joint-HM” (SM-JHM) means that a joint-HM is performed with inputs of [$\mathrm{source}=$ the colored image from stat-match, such as Fig. 3(e); $\mathrm{target}=$ the RGB image such as Fig. 3(a)]. Their results are presented Figs. 3, 6, and 7(h), which are better than the results from either stat-match or joint-HM. In fact, “stat-match then joint-HM” is overall the best among all color-mapping methods described in Sec. 4. Two examples of LUT-mapping colorization are given in Figs. 3 and 6(i). Figure 3(i) (an ideal case of LUT mapping) shows impressive colors; whereas in Fig. 6(i) the exhibits are noisy and distorted. The noise in the LUT-colorized image may be caused partially by the noisy reference image (taken at dusk) and partially by the pixel-based process during LUT table creation. In fact, many LUT-colored results (about 50% of 30 samples) are similar with Fig. 6(i). Some cases (e.g., Fig. 7) are not directly applicable to LUT colorization since no daylight reference image can be used. When using the LUT established in a different case at daytime (but aiming at different direction at night time), the colored results (not presented in this paper) usually appear worse. For surveillance or navigation applications where cameras move around (i.e., aiming at various directions), the LUT table may be created by using several pairs of night-time/daytime images taken on the camera moving along a path.¹²

Fig. 8

Night-vision coloring comparison: (a) and (b) and (e) and (f) are two samples of II and LWIR images in NV-set 2; (c) and (g) are the segmentation-based colorizations using histogram-matching, then statistic-matching; (d) and (h) are the channel-based color fusions of ($\mathrm{II}\oplus \mathrm{LWIR}$). Notice that there were no daylight RGB images available in NV-set 2.

The qualitative evaluations of six methods over three cases (shown in Figs. 3, 6, and 7) in NV-Set 1 are summarized in Table 1. Three categries of quality measurements are used for the qualitative evaluations, which are contrast, details, and colorfulness. The score of each measurement is rated from 3 to 1 to represent low, average, and high quality, respectively. Specifically, a high contrast means an adequate level of brightness and contrast, high details represent high clarity of detailed contents, and high colorfulness preserves more natural colors (i.e., closely resembles the daylight image). Columns 3 to 5 in Table 1 present the rated scores of three categories, where three scores in each cell corresponds to three cases shown in Figs. 3, 6, and 7, respectively. Notice that Fig. 7(i) is another sample of JHM and thus no score is given for LUT (shown $\times$ at the bottom row). The averaged scores are listed in the last column, where the quality rank is shown within a pair of curly brackets. It is clear that the quality order of colorization methods from the best to the worst: SM-JHM (stat-match then joint-HM), SM (stat-match), LUT, CBCF (channel-based color fusion), JHM (joint-HM), HM (histogram matching). The same acronyms of six colorization methods are used in Table 2.

Table 1

Qualitative evaluations (rated scores) of six methods over three cases (Figs. 3, 6, and 7) in NV-Set #1.(Rate: 1=high, 2=average, 3=low; ×=not applicable).

Table 2

The OEI (order) values of six methods over two cases (Figs. 3 and 6) in NV-Set #1. (The “qualitative rank” is recalculated with the rated scores of Figs. 3 and 6 in Table 1).

The quantitative evaluations using the OEI metric defined in Eq. (19) (refer to Sec. 5.2) are presented in Table 2 (corresponding to Figs. 3 and 6, respectively), where the ranks of metric values (1 for the largest OEI) are given within round parentheses. Keep in mind that the larger the OEI value of a colorized image the better its quality. According to the OEI values in Table 2, the quality order of colorized images in Fig. 3 from the best to the worst are (i), (h), (e), (d), (f), (g); and the quality order in Fig. 6 are (e), (h), (f), (i), (d), (g). To have an overall rank, the sums of the rank numbers in Figs. 3 and 6 are calculated and shown in Table 2. The rank of colorization methods (1 for the best) is given within the curly brackets. The order of colorization methods from the best to the worst: SM-JHM, SM, LUT, HM, CBCF, JHM. For a fair comparison, the averaged scores are recalculated with the rated scores of Figs. 3 and 6, which together with their qualitative ranks (same as Table 1) are exhibited at the far right column of Table 2. Both quantitative and qualitative evaluations support each other for the top three ranks, i.e., SM-JHM, SM, LUT. Statistical matching (SM) may cause color bias¹⁰ when the target (daylight) image is taken at a diffrent location from the source image. The joint-HM (JHM) can prevent (or reduce) color distortion when the source and target are similar in colors (see Fig. 2). On the other hand, JHM may increase color distortion if the source significantly differs from target (refer to the buildings and parking lots in Fig. 3). The JHM is typically combined with statistic-matching (i.e., SM-JHM), which makes the NV colorization better than any individual (either SM or JHM; see Fig. 3). Keep in mind the limitation of LUT method, i.e., both source and reference aiming at the same location. Although the performance of CBCF is poor, a realistic color fusion (as the source image) is always expected by other color-mapping methods. The OEI evaluations cannot be applied to Figs. 7 and 8 as no daylight images are available for the required reference images.

In NV-set 2, two pairs of multispectral images, image intensified (II) and LWIR, were analyzed by using color fusion and segmentation-based colorization methods as described in Sec. 4. The two input images are shown in Fig. 8(a) and 8(b) (provided by U.S. Army NVESD) and Fig. 8(e) and 8(f) (provided by the Netherlands TNO^9,13), respectively. Two input images in NV-set 2 were preregistered. The false-colored images (not shown here) were obtained by assigning II images to blue channels, infrared (IR) images to red channels, and providing averaged II and IR images to green channels. The segmentation was done in $l\alpha \beta$ space through clustering and merging operations. With the segment map (not shown here), the histogram-matching and statistic-matching were performed segment by segment in $l\alpha \beta$ space. After the training process was performed, the source region segments were automatically recognized and associated with proper target color schemes. The final colored images by segmentation-based colorization are shown in Fig. 8(c) and 8(g). From a visual examination, the colored images appear natural, realistic, and colorful. The details of segmentation-based colorization and experimental results (such as the colorized images with statistic-matching and histogram-matching methods) were presented in Zheng and Essock’s paper.¹⁰

A two-band channel-based color fusion [described in Eqs. (6)] was applied to the II and LWIR images [shown in Fig. 8(a) to 8(b) and 8(e) to 8(f)], and the results are illustrated in Fig. 8(d) and 8(h). The color-fusion results are reasonably good, especially in representing vegetation. Compared to the segmentation-based colorization results, the channel-based color fusion seems less realistic, such as the sky and roads. However, the processes of channel-based color fusion eliminates the need for segmentation and classification, and also reduces the color transforms. The processing speed is much faster than that of segmentation-based colorization. The LUT-mapping^12,13 method may not be directly applied to NV-set 2 since no daylight reference images are available (see Fig. 8). However, other mapping methods (e.g., joint-HM, stat-match then joint-HM) are applicable here by choosing a target image of similar scenery (such as Fig. 7), but those results are not presented here due to the limited space.

The qualitative (subjective) evaluations of NV coloration are based on casual visual inspections with three general categories. More qualitative measurements, subjective evaluations (by a group of subjects), and statistical analysis will be introduced in the future. The quantitative (objective) evaluations using the OEI require a reference (daylight) image. Thus we will continuously improve the OEI metric by relaxing the requirement of a reference image. We will further investigate color fusion, joint-HM, and SM-JHM methods, and their interactions for speed and visualization, as well as conduct more comprehensive comparisons.

7.

Conclusions

A set of qualitative and quantitative comparisons of NV colorization techniques is offered in this paper. We review a channel-based color fusion procedure; explore statistic matching, histogram matching, and LUT-based approaches; introduce new joint histogram matching and stat-match then joint-HM (SM-JHM) methods; and compare them with a segmentation-based colorization using both qualitative and quantitative evaluations. The quantitative evaluations using the OEI are consistent with the results of qualitative evaluations.

In summary, the segmentation-based colorization generates more colorful and more realistic night-vision images, but it requires heavy computations and thus is time-consuming. The channel-based color fusion gives reasonable coloring results and can be implemented for real-time applications. The LUT method also runs fast and yields a good result when the LUT table is properly established with direction independence. Statistic matching always works reliably and produces a stable colorization. Histogram matching often causes oversaturation and thus is more suitable for segmentation-based coloring. Joint histogram matching usually preserves the existing colors in a source image, which is not ideal when the source image (e.g., a false-colored image) is very different in color from the target image.

Overall, we recommend the “stat-match then joint-HM” (SM-JHM) method that effectively and efficiently provides impressive colorization. SM-JHM also demonstrates the best trade-off between image quality and speed over the methods explored. Keep in mind that the target image (a RGB image taken at daytime) used in all color-mapping methods (except for LUT) can be freely chosen with similar scenery, which may have a different resolution and requires no alignment.

Experimental results with multispectral imagery showed that the colorized images contain comprehensive information and vivid colors. The colorized NV imagery can significantly enhance the NV targeting by human users and will eventually lead to improved performance of remote sensing, night-time perception, and situational awareness.