1.

Introduction

Automated visual inspection systems are widely used in almost all fields of industry for inspecting goods and materials, including, e.g., paper, metal, wood, and textiles.¹ The use of various filtering techniques is common in automated visual inspection tasks, for example, by using various texture filters.¹ A number of examples can be found on this, for example, in rock surface characterization.² One growing area of the blemish detection is the manufacturing process of flat liquid-crqstal device displays³ and light-emitting device chips.⁴

The application area of this paper is the production testing of digital camera sensors. The number of pixels in high-resolution complementary metal oxide semiconductor camera sensors has increased rapidly from less than one million to eight million and over the past few years. In the camera manufacturing, it is necessary to inspect the quality of the camera by inspecting anomalies on the camera sensors that potentially are able to reduce the image quality given by the camera. Because of the increased pixel numbers, the quality inspection of the sensors has also become more time consuming. In addition, the huge production volumes have caused a need for extremely fast online inspection. The inspection systems must be able to accurately detect the anomalies, but not to find false defects in order to maintain as good a yield as possible.

In camera production testing, two most common defect types are defected pixels and blemishes. The defected pixels include hot pixels and dead pixels, which are typically significantly brighter or darker pixels than their neighborhood. In practice, dead pixels are more interesting in the current testing. A blemish is defined as a cluster of very mild dead pixels or a dark area caused by, e.g., dust or other objects inside a camera module. Both of these anomalies have a certain impact on the image quality given by the camera. The impact depends usually on the size and location of the defect on the sensor region; single dead pixels at the corner regions are not always critical, but the blemishes are typically more serious defects whose occurrence yields to the rejection of the camera.

The detection of the defects in the camera sensor is based on an image taken of a bright flat surface by the camera being tested. Because the dead pixel is like an impulse, it is relatively straightforward to detect by using an impulse-sensitive filtering approach. The blemishes, on the other hand, are much more difficult to detect because their edges are smooth and slow changes in the image signal, instead of impulselike dead pixels. Because of this difference, it is very difficult to detect these two defect types by using the same filter. Instead, two different methods are typically needed, which, in turn, increases the testing time. The images used in the production testing are typically raw sensor data that are not processed in any way, and consequently, several kinds of artifacts can be present in the data. Typically, the noise and vignetting artifact are the most serious challenges for the detection algorithms. The vignetting means a phenomenon of radial falloff of the image intensity from the image center.⁵ Hence, the edges and corners of the image become darker than the image central area. This kind of slowly changing average is able to cause problems to many blemish detection algorithms because the edges of the blemish regions are very similar as in the case of vignetting. Figure 1 shows an example of a typical production test image of resolution $2074\times 1558\text{pixels}$ . In this image, vignetting artifact is clearly visible.

Fig. 1

An example of a production test image with several blemishes and dead pixels (zoomed). There is also a clearly visible vignetting artifact in the image.

2.

Blemish Detection Algorithm

A straightforward method for the blemish detection would be to use, e.g., a differential filter that tries to detect the slowly ascending and descending signal values that could be interpreted as blemish edges. However, preliminary testing with real production test data revealed that these kinds of approaches are ineffective. This is because all these methods are sensitive to vignetting, which makes the blemish detection inaccurate. Also, the noise occurring in the image often causes difficulties in the blemish detection. In addition, separate filters must be used for the detection of dead pixels and blemishes, due to the different natures of these defects described in Section 1.

To overcome these drawbacks, we present a new, alternative solution for the blemish detection. The method has four major steps: (i) image size reduction by scaling, (ii) blemish detection by filtering, (iii) image subtraction, and (iv) thresholding. The method is based on the fact that the blemish regions are typically quite large compared to the actual image size. Consequently, by selecting an appropriate image-scaling method, image size can be significantly reduced, which also reduces the effect of noise. A scaling method used in this paper can be seen in Fig. 2 , where the sample pixels are selected from the image with $16\text{pixel}$ distance from each other. In this manner, a smaller image is formed by using an average value of every other pixel in this neighborhood to represent the pixel value in the scaled image. The scaling can be made in the same loop as the dead pixel detection and just save the scaling values into memory. When the scaling is made for every 16th pixel in both vertical and horizontal direction, the scaled image has only 0.4% of the pixel number of the original image. The size of the neighborhood was decided based on preliminary experiments, in which the effects of different neighborhood sizes on the noise levels were compared.

Fig. 2

The idea of scaling: the average value of every other pixel in $16\times 16\text{pixel}$ neighborhood is used to represent selected pixels in a new, smaller image.

As soon as the whole image is scaled, it can be used for blemish detection. A good approach for this would be to use an efficient mean or median filter to reduce the effect of the blemishes and then calculate a difference image between the filtered and original frame. In this paper, however, we present a more efficient way of making this by using a novel type of filtering approach that also calculates the difference image on the fly. The filtering approach aims at detecting the drops of intensity profile caused by a blemish as illustrated in Fig. 3 . The filter presented in Fig. 3 is applied on the rows $y$ of the frame $I(x,y)$ , and the filtering result is calculated by means of two median values calculated at distance $w$ from each other. The distance $w$ is selected to be large enough to cover the width of potential blemishes, but it should not be too long to avoid the possible disturbances caused by vignetting. The suggested value for $w$ is experimentally selected by inspecting the effect of value $w$ on the difference signal,

Fig. 3

(a) Intensity profile for line $y^{\prime }$ and (b) the principle of the filter.

The blemish regions can be extracted from the obtained difference image $I_{\text{difference}}$ by thresholding. One way of doing the thresholding is to use an adaptive threshold value that is dependent on the difference image, when it is more robust to differences in, e.g., exposure and sensitivity. This can be made by using a threshold proportional to the standard deviation of the whole difference image. The threshold value can be then written as $T=As$ , where constant $A$ is experimentally tuned. It is also possible to use a fixed threshold by using two fixed threshold values, upper and lower thresholds. If a pixel is over upper threshold, then its eight neighbor pixels are averaged, and if this average value is over lower threshold, then the pixel is marked as a blemish. By using this two-threshold technique, weaker blemishes can also be detected more effectively. The tuning of the thresholds can be based on, e.g., the intensity of the central area of the image, based on which it is easy to decide the constant threshold experimentally.

In the experiments, a test set containing 25 images taken with different sensors and varying exposure times, was used. Part of the images were significantly under- or overexposed. In most of these images, there were several blemishes. Once the algorithm was properly tuned, it could easily detect all the blemishes in the test set. In the tuning, selected sample images from a larger collection were used. Basically, the results show that once the algorithm parameters, namely, the threshold value, are properly tuned the algorithm is able to effectively detect the blemishes occurring in a certain sensor type. Also the blemish region edges could be localized accurately. An example of the blemish detection procedure can be seen in Fig. 4 .

Fig. 4

Filtering procedure for the example image presented in Fig. 1. From left to right: (a) Scaled image of resolution $64\times 48\text{pixels}$ , (b) filtering result obtained from the proposed filtering method, and (c) thresholded image.

3.

Discussion

In this paper, a novel method for blemish detection was presented. In this method, the image size is first significantly reduced with an appropriate scaling method that also effectively attenuates noise. After that, the blemish regions are extracted by the filtering method developed in this paper. This method makes the filtering and includes also the subtraction from the original frame. Hence, the memory requirement is minimized and separate calculation for difference image is not needed. This way, the blemish regions will have the highest values in the difference signal, and finally, it is possible to extract these regions from the difference image by thresholding.

Compared to the most other filtering-based methods, the presented approach has several benefits. First, it is not sensitive to vignetting or noise present in the image. Second, it is easy to implement the blemish-detection algorithm as a part of dead pixel detection. In practice, the proposed filter can detect both dead pixels and blemishes, because the dead pixels can be found by setting the filter width to three in the original image. If brighter values are also allowed [Eq. 3 can go negative], then it works also for hot pixels. Thus, the scaling operation necessary to the blemish detection can be a part of the dead pixel detection loop that is made pixel by pixel. Third, thanks to the very small image size used and the developed filtering method, the computational time required by the algorithm is very low. Especially, the computational time for blemish detection can be made very low, the current implementation of blemish detection (including scaling) requires only 7% of the time required for the dead pixel detection by using the same filter. In general, the proposed method makes the overall testing time of both dead pixels and blemishes greatly shorter on the production line than that of previously used method that detects merely the dead pixels.