1.

Introduction

Various technologies and methods for improving image quality in low-light environments have been proposed. One of these approaches involves increasing the sensitivity of visible band (VIS) signals.^1–4 Another approach involves combining VIS and infrared (IR) light by obtaining luminance information from the sensor response of IR and synthesizing color information from the sensor response of VIS light so that color image representation can be used, even in dim environments.^5–13 However, neither of these methods can be used to obtain color information in the complete absence of VIS light. To solve this problem, the usual approach is color estimation of monochrome images. This is an image restoration technique based on the prior color information obtained by either one of the following options: (1) color specified by the user,^14–16 (2) arbitrary color images to estimate color of monochrome images,^17–19 or (3) machine learning using sample images.^20–22 The colorization technique, a method of obtaining a colorized IR image by using a color image as initial information, has also been proposed.^23–25 However, all these colorization techniques require to specify color information or to analyze many color images in advance.

It has been proposed to reproduce VIS color by the multispectral images in IR region.²⁶ Identifying spectral reflection in IR related to that in VIS, pseudocolor representation was obtained only from the three IR response (basic IR color).^27,28 However, basic IR color was an approximation that could not achieve the high color reproducibility of VIS color.

In this study, we propose an image processing method for improving color reproducibility by effectively utilizing VIS light in addition to basic IR color in dim (enhanced IR color), particularly for the objects that are not suitable for basic IR color. The enhanced IR color is superior to basic IR color because it is comparable to basic IR color in dark and better in dim and bright illuminance conditions by utilizing VIS. Therefore, the enhanced IR color is a unique method providing favorable color reproducibility in dark, dim, and bright illuminance conditions without using prior color information. It should be noted that in this study, IR refers to near-IR, approximately in the range of 700 to 1000 nm. As for the illuminance level, the meanings of “bright,” “dim,” and “dark” are “enough VIS light,” “low VIS light,” and “absence of VIS light,” respectively.

2.

Basic IR Color

2.1.

Image Processing for Basic IR Color

Basic IR color, which has been proposed previously, is a technique for colorizing an IR image close to the VIS color by performing color processing based on the weak relationship between the reflection characteristics of the objects in the VIS and IR range. Specifically, by attaching a correspondence between the three primary colors of VIS and the multiple wavelengths of IR, it is possible to estimate the VIS color based on the signal intensity of the multiple IR wavelengths and perform color representation.^29–31 Image processing methods for IR color are similar to those for VIS color, including white balance (WB), color correction (CC), and noise reduction (NR). An example of image processing for the basic IR color is shown in Fig. 1. It should be noted that image processing can be performed in various color spaces, such as RGB, YCbCr, and CIELab.

Fig. 1

Example of IR color camera and image processing flow.

WB, CC, and NR are conventional image processing methods used for VIS as well. However, because the spectral reflections in the IR region are more gradual than those in the VIS region,³² it is necessary to provide a specific adjustment in the image processing of IR color more than VIS color. The IR color-specific counterparts are as follows.

2.1.1.

White balance

WB means here the global adjustment of the color intensities to make a white object appear white using IR illumination. Because the balance between IR wavelengths varies significantly depending on the light source irradiated on the objects and the sensitivity of the image sensor, the ratio of the IR signals from the imaging sensor can be largely diversified. Therefore, the WB gain must be applicable to a wider range of values (e.g., 0.125 to 8.0) than those for the VIS response. In addition, when a value of $<1.0$ is set for the WB gain, specific processing is required to avoid the so-called high-luminance coloring phenomenon, which causes unwanted coloring in the saturated portion of high luminance.

2.1.2.

Color correction

Here, CC means a localized color adjustment allocating IR signals to VIS color. As for the characteristics of the materials used as objects, the spectral characteristics in the IR wavelength region are generally less sensitive than those in the VIS wavelength region, and the relative changes between wavelengths are more gradual. To visualize such feature representation, it is necessary to provide relatively strong correction effect, such as a variable range of $\pm 4.0$ for the RGB $3\times 3$ matrix coefficients. The colors are adjusted through matrix correction to the approximate VIS color.

2.1.3.

Noise reduction

Owing to the strong correction gain, the NR function plays a crucial role along with the CC function. Maintaining minimum noise while optimizing image quality, including sharpness and edge expression, is essential. In this system, two types of filters, space NR (Gaussian type smoothing filter) and time axis NR (frame addition) are combined especially in low IR illumination where the noise level is high. The purpose of basic IR color is providing color information and it has a trade-off relationship with NR; therefore, an appropriate balance should be set between both.

2.2.

Example of VIS and Basic IR Color

Figure 2 shows an example of the VIS and basic IR color. The objects surrounded by blue frames (tomatoes, ink bottles, and paper clips) reproduce color relatively well, whereas the Macbeth chart surrounded by red frames shows low color reproducibility, thereby suggesting that the accuracy of color reproducibility varies depending on the objects.

Fig. 2

Sample images of VIS and basic IR color: (a) VIS color in bright illuminance (30 lux) and (b) IR color in dark illuminance (No VIS used).

3.

Enhanced IR Color

In this section, we describe an imaging process to enhance IR color utilizing RGB-3NIR six-channel multispectral signals. Although basic IR color uses only IR light assuming complete darkness, in practical applications, such as surveillance, it is rare that there is no VIS light. There are many dim situations where there is a small amount of VIS light. In such cases, instead of providing basic IR color by cutting off all VIS light, we examined a method for improving color reproducibility by effectively utilizing weak VIS signals.

The camera system used in this study is illustrated in Fig. 3. The multispectral camera (NLX-PH001-C) was selected because it is designed to provide VIS and basic IR color. The camera has 3CMOS with a spectrum prism as shown in the transmittance characteristics in Fig. 3. Each CMOS detects different VIS and IR bands: the first CMOS for R and IR1 (810 nm), the second CMOS for B and IR2 (870 nm), and the third CMOS for G and IR3 (940 nm). The camera works as a VIS camera when attaching an IR cut-filter and as a basic IR color camera with a VIS cut-filter. In this exercise, two of these cameras were used because one camera cannot provide VIS and IR six-channel multispectral signals. One (CAM1) is equipped with an IR-cut filter to transmit only R, B, and G. The other without a filter (CAM2) transmits R and IR1, B and IR2, G and IR3. The angles and distance of both cameras are adjusted with using calibration software to avoid pixel shifts. There are two illuminators for the VIS and IR. The VIS illuminator is variable for adjusting the darkness. The IR illuminator is constant and irradiates IR1, IR2, and IR3 light to the target.

Fig. 3

Evaluation system of the enhanced IR color camera: (a) the total system, (b) schematic channel responses for CAM1 and CAM2, (c) normalized spectral intensity of IR illumination (Nanolux Co. Ltd., multi-IR illuminator), and (d) that of VIS illumination (CCS Inc., XX-SC28FS55-P1).

The image processing algorithm for enhanced IR color is shown in Fig. 4. Three unique issues of enhanced IR color must be considered: The first issue is the difference in the spectrum characteristics in the VIS and IR ranges; an optimal image cannot be obtained by simply summing the VIS and IR colors but has to be processed in VIS and IR ranges separately. The second issue is the dynamic transition of the optimal composite ratio of VIS and IR images based on illuminance level; for higher illuminance level, VIS is more reliable than IR color. Therefore, the optimal composite ratio needs to be calculated in the algorithm seamlessly and automatically. The third issue is the relatively weak spectral reflections in the IR region compared with those in the VIS region. Therefore, a strong parameter is required in WB and CC in IR image processing as described in Sec. 2.1. To compensate the noise caused by the strong parameters, a spatial quality signal such as SN (contrast-preferred signal) is separately calculated and added to the signals generated with emphasis on color expression such as hue and Chroma (tonal-preferred signal) in VIS and IR.

Fig. 4

Algorithm for enhanced IR color and three main issues.

3.2.

Solution to Issue 2: Optimal and Automatic VIS and IR Composition Under Various Illuminance Levels

As the lighting environment changes, the optimal composition ratio of VIS and IR colors changes as well. It is necessary to automatically transit the optimal composition of the VIS and IR colors according to the external lighting environment. Although linking parameters by installing a separate illuminance sensor could be considered, the price and complexity of such a method would be impractical. Therefore, we develop a method for setting the optimal composition ratio without an additional sensor by referring to the auto gain value from the auto exposure (AE) function of the camera.

The basic idea of the method is that (1) as the auto gain value for VIS decreases, the composition ratio of VIS decreases and that of IR increases, and (2) as the auto gain value for IR decreases, the composition ratio of IR decreases as well.

The composition is performed on the signals after WB correction, CC, and gamma conversion, as shown in the “VIS and IR composition” presented in Fig. 4. The composition formula is a simple weighted sum operation, which is expressed as follows:

Eq. (4)

$$\left[\begin{array}{c}Y\\ C_{bl}\\ C_{rl}\end{array}\right]=\left[\begin{array}{c}{w}_{vY}{Y}_v^{\prime }\\ w_{vC}{C}_{bv}^{\prime }\\ w_{vC}{C}_{rv}^{\prime }\end{array}\right]+\left[\begin{array}{c}{w}_{iY}{Y}_i^{\prime }\\ w_{iC}{C}_{bi}^{\prime }\\ w_{iC}{C}_{ri}^{\prime }\end{array}\right],$$

where $Y_{v^{\prime }}$, $C_{b{v}^{\prime }}$, and $C_{r{v}^{\prime }}$ are the color reproducibility signal after development from VIS response. $Y_{i^{\prime }}$, $C_{b{i}^{\prime }}$, and $C_{r{i}^{\prime }}$ are the color representation signal after development from IR response. $w_{vY}$ is the weight coefficient for luminance signal $Y_{v^{\prime }}$ after development from VIS response. $w_{vC}$ is the weight coefficient for chromaticity signals $C_{b{v}^{\prime }}$ and $C_{r{v}^{\prime }}$ after development from VIS response. $w_{iY}$ is the weight coefficient for luminance signal $Y_{i^{\prime }}$ after development from IR response. $w_{iC}$ is the weight factors for chromaticity signals $C_{b{i}^{\prime }}$ and $C_{r{i}^{\prime }}$ after development from IR response.

At next, the weight coefficients for the VIS light response, $w_{vY}$ and $w_{vC}$, are calculated using the gain values for the VIS response

Eq. (5)

$$w_{vY}=w_{vC}=F_{DQv}(D_{av}),$$

where $D_{av}$ is the auto gain value for VIS response (dB). $D_{ai}$ is the auto gain value for IR response (dB). $F_{DQv}$ is the evaluation function of VIS image quality through VIS response gain.

As the auto gain value of the VIS light decreases, the ratio of VIS light should decrease as well.

Finally, the weight coefficients $w_{iY}$ and $w_{iC}$ for the IR response are calculated using the gain values for the VIS and IR responses

Eq. (6)

$$w_{iY}=F_{DvSY}(D_{av})\times F_{DQi}(D_{ai}),$$

Eq. (7)

$$w_{iC}=F_{DvSC}(D_{av})\times F_{DQi}(D_{ai}),$$

where $F_{DvSY}$ and $F_{DvSC}$ are the evaluation function of each signal complement intensity through VIS response gain, $F_{DQi}$ is the evaluation function of IR response image quality through IR light response gain.

$F_{DvSY}$ and $F_{DvSC}$ represent the strength of the IR signal that is allowed to complement the signal obtained from the VIS response when it decreases, and they should be set such that they increase as the gain value for the VIS response decreases.

Figure 5(a) shows an example of the evaluation function. Complementation through the IR signal is set to start at a slightly higher gain, which is intended to keep the VIS color as intact as possible. If the color approximation of the IR color is higher, complementation through the IR signal can be set to start at a lower gain.

Fig. 5

Example of evaluation functions for color composition: (a) evaluation function from VIS response and (b) evaluation function from IR response.

$F_{DQi}$ is set to reduce the weighting of the IR color when the signal from the IR light response decreases, thereby indicating that the IR color is no longer reliable, as shown in Fig. 5(b).

3.3.

Solution to Issue 3: Optimal Composition of Tonal- and Contrast-Preferred Signals

The final image is constructed by applying separately generated contrast- and tonal-preferred signals obtained by the method explained in Sec. 3.2.

The advantage of the contrast-preferred signal is to combine the full signals from the two cameras. Figure 6 shows the detailed processing configuration for combining the tonal- and contrast-preferred signals. As shown in Fig. 6, the contrast-preferred signal is used as the base signal. The difference between the tonal- and contrast-preferred signals is used as an additional signal to ensure the quality of chromatic expression that is finally added for output. The spatial filter for NR processing applied to the tonal-preferred signal or additional signal is an edge-preserving low-pass filter that reduces the high-frequency component caused by noise, but it retains the edge components as much as possible. Finally, the two signal spatial filter was used as an effective process for achieving differential NR.

Fig. 6

Detailed processing configuration for tonal- and contrast-preferred signal composition.

The image processing for the tonal- and contrast-preferred signal composition process is shown in Fig. 7.

Fig. 7

Example of images in the tonal- and contrast-preferred composition process.

4.

Experimental Evaluations

The images obtained using the enhanced IR color system in bright (30 lux), dim (0.7 lux), and dark (0.15 lux) illuminance are compared with basic IR and VIS in Fig. 8. Enhanced IR color provides better color reproducibility of the Macbeth chart (see the red boxes) in dim light than basic IR color while achieving color reproducibility comparable to basis IR in dark and VIS in bright illuminance.

Fig. 8

Enhanced IR color, basic IR color, and VIS color by illumination level.

Figure 9 shows the quantitative evaluation of color reproducibility of the Macbeth chart. We select 12 colors in the second and third rows from the top of the standard, where it was difficult to reproduce the colors using basic IR color. Because the main purpose of the study is to reproduce VIS color using the multispectral signals, the chromatic variance ($\mathrm{\Delta }{C}_{ab}$) is measured, using the distance from the correct color of VIS in bright (30 lx) illuminance in the $a-b$ dimension of the CIELab color space³³

Fig. 9

Chroma reproducibility evaluation of basic IR color and enhanced IR color of Macbeth chart: (a) 12 colors of Macbeth chart used for the evaluation, (b) distances from the correct color (VIS color in bright) on $ab$-space on average of 12 colors, and (c) those of each color.

The shorter the distance, the more reproducible the chroma. Because basic IR color does not utilize VIS light, there is always a large discrepancy from the VIS color regardless of the illuminance. In contrast, enhanced IR color provides more reproducible colors than basic IR color, not only in bright but also in dim light.

To verify the reproducibility of other objects, the same analysis was performed on the three ink bottles (see the dotted blue boxes in Fig. 8), which achieved relatively good color reproducibility using basic IR color (Fig. 10). Comparing the color reproducibility of basic IR color with the Macbeth chart presented in Fig. 9, the ink bottles indicate slightly better color reproducibility (66 vs. 47). Comparing the basic IR color with the enhanced IR color, the latter is superior on average at all illumination levels for all three colors, although there are cases where the color expression is better in darker illuminance, e.g., the green bottle under 0.7 to 5.1 1ux. Thus, it was determined that enhanced IR color can achieve more reproducible colors than basic IR color, particularly for objects with poor color reproducibility.

Fig. 10

Chroma reproducibility evaluation of basic IR color and enhanced IR color of ink bottles: (a) three colors of ink bottles used for the evaluation, (b) distances from the correct color (VIS color in bright) on $ab$-space on average of three colors, and (c) those of each color.

5.

Conclusion

We proposed an image processing method for improving color reproducibility compared with that of basic IR. By utilizing a total of six channels (three VIS and IR wavelengths each), we confirmed that the color reproducibility of enhanced IR color exceeded that of basic IR color under all light conditions. We also proposed an algorithm that takes advantage of both VIS and IR colors to automatically transition the optimal composition of VIS and IR colors using auto-gain values that can be easily obtained from an ordinary camera. We confirmed the superiority of enhanced IR color to basic IR color for various objects, particularly for objects with poor color reproducibility in basic IR color.