2.1.

Lidar

The lidar features utilized in our machine learning technique include unidirectional reflectivity and range. Reflectivity is used to characterize the material, and range is used to estimate the observation angle of the material surface. The direct detection pulsed lidar sensor utilized in this work operates at the $1.55\text{-}\mu \mathrm{m}$ wavelength and uses a linear mode avalanche photodetector. The system transmits a 5-ns full-width at half-maximum laser pulse which strikes and scatters opaque surfaces. The intensity of the backscattered laser energy is captured by the photodetector and digitized by a receiver. The time elapsed between the transmitted and reflected pulses is used to calculate range. Multiple range measurements across a fraction of a surface can be used to estimate angle of incidence. The peak of the backscattered pulse is used to estimate reflectivity.

As shown in Fig. 1(a), active sensors are typically dominated by unidirectional radiance represented by Eq. (1) with ${\theta }_r={\theta }_i$ and ${\varphi }_r={\varphi }_i$, which we denote as $\theta$ and $\varphi$, respectively. However, the receiving detector is polarization insensitive; therefore, only the $s_0$ component of ${\mathbf{L}}_{\mathbf{r}}$ is measured. Furthermore, we assume nondiagonal elements in the first row of the Mueller matrix for our data to be zero. This assumption is supported by experimental measurements^28,29 of diverse materials which show that nondiagonal Mueller matrix elements of most opaque surfaces that might be observed in a remote sensing application are approximately zero. Using the stated simplifications, Eq. (1) is approximated for our lidar system as

Fig. 1

Depiction of common geometries for radiometric sources in (a) active (unidirectional) and (b) passive (specular, diffuse, and self-emission) imaging.

Due to practical complications in measuring $E_i$ in Eq. (3), $f_r$ is defined in alternative form as

2.2.

Passive Polarimeter

The polarimetric feature, DoLP, is captured using a cooled Polaris 640 LWIR Imaging Polarimeter, manufactured by Polaris Sensor Technologies, Inc.³² The sensor has an operating wavelength of 7.5 to $11.1\mu \mathrm{m}$ and up to a 120-Hz frame rate. The Polaris 640 system is equipped with a fixed polarizer and rotating retarder imaging polarimeter, which takes measurements of linear polarization oriented at 0 deg, 45 deg, 90 deg, and 135 deg such that $L_{0\mathrm{deg}}$, $L_{45\mathrm{deg}}$, $L_{90\mathrm{deg}}$, and $L_{135\mathrm{deg}}$ are scalar measurements of $\mathbf{L}$. The measurements are combined using the modified Pickering’s method⁵ described as

A common characterization of polarization in passive polarimetric imaging is DoLP, which is calculated from $\mathbf{L}$ as

The fundamental properties of polarization suggest that polarimetric measurements could be useful features for material classification, specifically in discriminating rough and smooth surfaces.^9,36 This is typically explained by representing the texture of the surface as multiple microfacets with orientations following a random distribution. The angle-dependent polarization from each microfacet is incoherently summed when simultaneously observing multiple microfacets of a rough surface, resulting in an unpolarized signal. Conversely, smooth surfaces maintain a consistent orientation across the surface and therefore preserve the polarimetric signal.

2.3.

Data Representation

This paper advances material classification by utilizing the feature set consisting of measurements of lidar and passive polarimetric sensors both characterized over a well-defined set of observation angles. The number of unique observation angles and the specific angles utilized are expected to significantly affect classification performance. For example, from Fresnel reflectance theory, DoLP is known to increase as observation angle relative to normal increases.³⁷ Concerning the mBRDF angle dependence, perfectly diffused Lambertian surfaces have uniform $f_r$ for all angles; however, realistic surfaces typically have specular components with higher values within the normal incidence specular lobe.³⁰ We assume the observation angle can be determined by estimating surface orientation relative to normal using lidar 3-D point-cloud imagery. The observation angle is represented as $\theta$ and is restricted to be in the monostatic plane of incidence such that $\varphi =0\mathrm{deg}$. Furthermore, in many applications, multiple observation angles can be measured on a single material surface, due to a moving platform or moving object. The features are jointly represented by feature vector $\mathbf{X}$ as

3.1.

Material Classification

Since both $f_r$ and DoLP are expected to have consistent and repeatable measurements in most situations, a supervised learning algorithm is considered for the hybrid sensor material classification in this paper. In supervised machine learning, labeled sample data is used offline to model the mapping between input examples and the known output classes.³⁸ We utilize features measured against laboratory data of a diverse material dataset to train the supervised classifier in identifying material type. The key idea of supervised learning is to estimate a decision boundary, which separates each class from one another based on the training data. We propose using a support vector machine (SVM) for classifying material type due to the proven success of this classifier in similar applications such as hyperspectral imaging for land cover classification and target detection.^39,40 However, we believe advanced classifiers could be designed, based on the proposed technique (i.e., hybrid sensing with known viewing orientation), that optimize performance for a specific application. The SVM presented in this paper demonstrates the general application of material classification.

The SVM classifier tries to find the optimal separating hyperplane that maximizes the margin between the closest training samples of each class. The hyperplanes are typically formed in high-dimensional space using kernel transformation functions;⁴¹ and boundary pixels (i.e., support vectors) are utilized to create a decision surface.⁴² Therefore, SVM classifiers are inherently binary classifiers designed to solve two-class problems. A collection of SVM classifiers must be implemented to separate multiple classes. Multiclass designs include one-versus-all (one SVM classifier for each class) and one-versus-one (one SVM classifier for each pair of classes). The SVM classifier is a particularity popular solution for machine learning when there are a limited number of training samples available,⁴³ which is typically the case in nonconventional imaging, such as hyperspectral, polarimetric, and lidar. The implementation, parameter selection, and classification accuracy of material classification for our proposed hybrid sensing system are presented in Sec. 4.

3.2.

Notional System

The proposed hybrid sensing architecture is beneficial to a multitude of machine learning applications, such as automatic target detection, land cover classification, autonomous driving, and machine vision in manufacturing. The actual system parameters of the lidar and passive polarimetric sensors should be carefully selected to optimize the performance for the specific application. For example, commercially available lidar systems designed for autonomous driving currently utilize high scanning rates and a large field-of-view, requiring high repetition rate lasers with moderate power and $\sim 200\text{-}\mathrm{m}$ maximum distance.^44–46 In contrast, scanning linear-mode lidar in 3-D mapping remote sensing applications typically requires a higher power laser and operates at an altitude of $\sim 1000$ to 5000 ft,⁴⁶ with operating ranges of $\sim 1\mathrm{km}$ or greater. In this section, we demonstrate the feasibility of the proposed architecture by presenting a notional implementation for a remote sensing application.

To support our notion of hybrid sensing, a tactical demonstrator is fully assembled using the commercially available passive polarimetric imager manufactured by Polaris Sensor Technologies, Inc. as described in Sec. 2.2, and a custom lidar system owned and operated by the Air Force Research Laboratory (AFRL) at Eglin Air Force Base. Parameters for the demonstration are shown in Table 1. The system is operated to capture imagery at $\sim 1.5\mathrm{km}$ from a 25-m tower. A flat white painted aluminum $1.22\mathrm{m}\times 1.52\mathrm{m}$ panel is placed in a predominately natural scene at a 1.469-km slant range and 40-deg observation angle. Example imagery from the demonstration is shown in Fig. 3. At this range, there are $\sim 88$ and 12 pixels on the panel with the lidar and passive systems, respectively. For this application, the passive system is designed to have a larger field-of-view to locate possible objects-of-interest, and the lidar is cued to image specific areas with high resolution. The presented notional hybrid system demonstrates the feasibility to capture imagery using a tactical system in a relevant application.

Table 1

Parameters of hybrid passive polarimetric and lidar demonstrator system.

Fig. 3

Imagery from the hybrid sensor demonstration showing (a) $f_r$ from the lidar system and (b) DoLP from the passive polarimetric imager, with the flat white painted aluminum panel circled in red.

If the proposed architecture is utilized in applications where long ranges or adverse weather conditions are present, the lidar measurements must be corrected to compensate for atmospheric attenuation and signal loss using a radiometric model. The first technique to mitigate this issue is choosing an operating wavelength of the laser to be within a high transmission window. In addition, we suggest utilizing a popular radiometric model, such as MODTRAN⁴⁷ or LEEDR,⁴⁸ as well as current meteorological data, to correct for atmospheric effects. The passive polarimetric signal DoLP is not altered due to signal attenuation, but sources of noise such as diffuse reflected LWIR radiance could affect the polarimetric signal. In this paper, we do not attempt to correct measurements taken in adverse conditions and long ranges. Instead, we limit our measurements in this paper to close range under ideal conditions and then introduce a generic error source into the test database when evaluating the classification accuracy (discussed in Sec. 4.4). The error term represents effects of long range imaging and atmospheric conditions (or possible errors resulting from the correction of those effects). Adding error to our data alters the signal-to-ratio (SNR), which is varied to represent multiple degrees of accuracy that may be expected. Basically, longer ranges and more difficult imaging environments are expected to reduce the SNR, and we evaluate performance against varying amounts of SNR.

4.1.

Dataset

To our knowledge, there are no lidar datasets with LWIR passive polarimetric imagery available to evaluate the performance of material classification algorithms. Therefore, an experiment is conducted to obtain a unique characterization of a diverse set of materials with both active and passive polarimetric infrared imaging systems. The experiment is conducted to collect $f_r$ and DoLP of 34 materials imaged at eight observation angles. The sample materials consist of painted aluminum panels (of various colors and gloss), painted tile thinset (of various colors and texture), naturally occurring objects (e.g., leaves, pine needle, and bark), asphalt, concrete, brick, rubber, metal, roof shingle, plywood, plexiglass, and cardboard, as shown in Fig. 4. The diverse set of materials are categorized into 19 classes, which are labeled as classes $a$ through $s$. Measurements of each material are analyzed in Sec. 4.2, and class groupings are used for classification in Sec. 4.4.

Fig. 4

Materials utilized in experiments separated into 19 classes labeled $a$ through $s$. For each material, a picture and name are shown (there are two green and two black painted aluminum samples made by different paint vendors).

Each sample is placed on a rotation stage controlled by an articulating tripod which has the ability to pan and tilt via computer-controlled instruction. Samples are imaged at angles 0 deg to 70 deg in 10-deg increments where 0 deg is normal incidence (as determined by a mirror) and $\varphi$ is held constant at 0 deg. The entire scene remains static for each iteration of imaging. The scanning lidar system captures pulse intensity at each pixel of the image by measuring the peak power of the backscattered pulse. A region of interest (ROI) is manually selected in the lidar imagery to represent approximately the same portion of the material for all angles, as shown in Fig. 5. The ROI is selected to include all of the sample surfaces except for areas near the edge. The ROI consists of at least 1800 pixels at normal and 250 pixels at 70 deg. The measurements are taken in a controlled laboratory setting at a distance of $\sim 9\mathrm{m}$. Measurements are also taken against calibrated Spectralon panels with ${\rho }_{\mathrm{DHR}}$ accurately measured at the $1.55\text{-}\mu \mathrm{m}$ wavelength. Using the mean power measurements of the materials and Spectralon panels, $f_r$ is calculated using Eq. (5).

Fig. 5

Experiment setup showing sensor measurements of (a) lidar normal to material surface, (b) lidar at 70 deg, (c) LWIR intensity at normal, and (d) LWIR intensity at 70 deg. ROIs are shown in blue for the active system.

The entire experiment process is repeated using an LWIR polarimeter in place of the lidar system. In order to capture the emissive properties of the material, a heating element is utilized to maintain an $\sim 100°\mathrm{C}$ surface temperature. The passive polarimeter measures the Stokes column matrix, as described in Eq. (6) (example imagery is shown in Fig. 5). ROIs are manually selected and consist of at least 3000 pixels at normal observation angle and 650 pixels at 70 deg. Finally, DoLP is calculated using Eq. (7). More details of the experiment setup and methodology have been recently published.²⁴

The sample mean $\overline{X}$ and standard deviation ${\sigma }_{\mathrm{SV}}$ of the pixel values within each ROI are calculated to statistically represent the experiment measurements as random variables. For simplicity, both $f_r$ and DoLP are approximated as Gaussian distributions. The feature set of Eq. (9) is formed using the experiment measurements of each material described as

Fig. 6

Lidar $f_r$ mean measurement (within ROI) versus observation angle for materials in classes (a) a–c, (b) d–f, (c) g and h, (d) i–k, (e) l–n, and (f) o–s. Standard deviation is shown as error bars for each data point.

Fig. 7

Passive polarimeter DoLP mean measurement (within ROI) versus observation angle for materials in classes (a) a–c, (b) d–f, (c) g and h, (d) i–k, (e) l–n, and (f) o–s. Standard deviation is shown as error bars for each data point.

4.2.

Data Analysis

Next, we analyze the dataset obtained with the hybrid sensor experiment. Inspection of $f_r$ in Fig. 6 shows the sample mean of materials with semigloss or glossy paint have extremely large $f_r$ near normal (due the specular lobe of the lidar geometry) and low diffuse $f_r$ at other angles. The $f_r$ of all other materials tends to vary slowly with observation angle because the backscatter energy is mostly diffused reflectance. Dark color paints (i.e., green, black, and camouflage) have much lower $f_r$ than light colors (i.e., tan, white, and gray) because the darker colors absorb some of the laser energy. Additional groups of materials with considerably low reflectance include asphalt, rubber, and rusted steel. Materials painted light colors and brick have the overall highest $f_r$. The natural materials, roof, concrete, cement block, cardboard, plywood, and plexiglass have similar $f_r$ that is typically more than dark paints but less than light paints.

According to Fresnel polarization theory,²⁶ the magnitude of linear polarization is zero at normal observation angle and increases as a function of angle and refractive index of the material. For rough surfaces, the polarization is degraded as the signal from each microfacet is incoherently summed.³⁴ In our dataset, DoLP is approximately zero near normal observation angle and increases with angle for almost all materials (resulting in a $-s_1$ and $+\mathrm{DoLP}$). The only exception is plywood which has a reflected component that is prevalent at observation angles less than 20 deg ($+s_1$ and $+\mathrm{DoLP}$). Aluminum with light or dark paint color has the highest DoLP due to very smooth surfaces. Natural materials have the lowest DoLP, due to rough surfaces. Likewise, the smooth, medium, and rough textured thinset has DoLP inversely proportional to the surface roughness. Many of the measurements within a class maintain very similar signatures. For example, all materials of the semigloss light painted aluminum class (class $e$) have approximately the same polarimetric signal for all angles [as shown in Fig. 7(b)]. However, in comparison to the $f_r$ measurements, DoLP appears to be less diverse between classes. For example, class $e$ is very similar to classes $d$ and $f$. Therefore, classification may be more difficult with DoLP. Overall, the combined dataset is seen to agree with reflectance and polarization theory.

As previously discussed, the standard deviation represents material variation due to surface texture. In lidar imagery, standard deviation is relatively small compared to the mean, with the exception of the glossy and the camouflage painted aluminum panels. The glossy paints have a nonuniform specular spot at the center of the material near normal observation angles. The camouflage sample has three different paint colors within the ROI which causes a high standard deviation. As anticipated, the standard deviation of DoLP is highly correlated with the surface roughness (i.e., rough and smooth surfaces have high and low standard deviation, respectively)³⁴ and mixed material types. For example, thinset with rough texture has higher standard deviation than the smooth thinset. Similarly, oak leaves and rusted steel have significantly higher variance due to the diverse materials within the ROI (i.e., colors of leaves, rust deposits on steel, etc.).

4.3.

Implementation of Supervised Learning

The complete dataset which is composed of sample mean and standard deviation presented in Figs. 6 and 7 is utilized to generate a database for supervised machine learning and classification performance evaluation. The initial database contains 34 row vectors, where each $1\times 16$ row vector contains $f_r$ and DoLP measured at eight observation angles, as described in Eq. (9). For each of the 34 material samples, 100 observation vectors are generated using the sum of sample mean $\overline{\mathbf{X}}$ and randomly distributed Gaussian noise ${\eta }_{\mathrm{SV}}$ characterized by the material’s variance ${\sigma }_{\mathrm{SV}}$, as described in Eq. (10). The entire database is organized as a $3400\times 16$ matrix, to represent an ensemble of measurements of the material. Class labels are assigned to each observation following the class grouping $a$ through $s$ as indicated in Fig. 4. The generated database represents intrinsic variation due to surface texture and inconsistent material properties across the sample surface with no measurement noise added (e.g., rust, discoloration, grain, nonuniform mixtures, etc.). To address measurement noise, we introduce a separate noise component, which is described in Sec. 4.4.2.

We propose the use of SVM to implement material classification, as discussed in Sec. 3.1; however, we emphasize the hybrid sensing architecture is prevailing over single modality sensing while using any one of an assortment of supervised machine learning techniques. Therefore, in addition to SVM we also implement decision tree,³⁸ discriminant,⁴⁹ Naïve Bayes,⁵⁰ $k$-nearest neighbors ($k\mathrm{NN}$),⁵¹ and neural network⁵² to prove the benefit of hybrid sensing. All classifiers are implemented using either the Statistics and Machine Learning or Deep Learning toolboxes from MATLAB.⁵³ First, the database is loaded into the Classification Learner tool in MATLAB and the option to partition into five disjoint folds is selected. This option utilizes four folds for training and one fold is used for testing. To reduce classification variability, five rounds of cross-validation are performed using different partitions, and the validation results are averaged to obtain the final classification accuracy. Next, each of the six classifier techniques is individually selected within the tool. Parameters of each classification method are iteratively adjusted as shown in Table 2. All combinations of the parameters are exhaustively exercised, and the optimal result is utilized in the final accuracy metric for each implementation. Please note: best-performing parameters within the listed parameter-space change depending on the number of viewing angles (i.e., features), SNR, and classes of the dataset. Furthermore, future implementations could utilize automatic selection of the parameters via optimization tools provided by MATLAB to optimize the classifier for specific applications. Finally, the classification learner tool allows the user to select a subset of the features in the database to utilize in training and testing. In the following section, we present results from the experiment using various combinations of viewing angle measurements.

Table 2

Parameter space explored for each classification technique.

4.4.

Performance Evaluation

To fully demonstrate the added benefit of multisensor material classification, supervised techniques are utilized with features of individual sensors as well as the proposed hybrid system. We also experiment with multiple combinations of observation angles. First, results of a single observation angle using $f_r$ only, DoLP only, and hybrid features are evaluated without measurement noise added. Then, performance using a single observation angle is evaluated with varying levels of noise added. Finally, results using multiple observation angles with noise are presented.

4.4.1.

Single observation angle without measurement noise

Measurements at a single observation angle, $f_r({\theta }_1)$ and $\mathrm{DoLP}({\theta }_1)$, are utilized and ${\theta }_1$ is varied from 0 deg to 70 deg. Total classification accuracy, calculated as the number of observations correctly classified out of the total number of observations, is determined for each angle. As shown in Fig. 8(a), classification with $f_r$ has consistent performance for all angles and DoLP improves as ${\theta }_1$ increases. The result matches expected performance based on Fresnel reflectance, where DoLP increases with angle and material classes become more distinct as observation angle increases. The highest classification accuracy obtained in this experiment is 83.6%, which occurs at ${\theta }_1=70\mathrm{deg}$. By utilizing multiple features, classification accuracy is increased by 44.5% compared to lidar only, and 32.3% compared to passive polarimetric only; however, since a standalone passive polarimeter cannot determine observation angle without lidar point-cloud information, the DoLP only classifier is still dependent on the ranging information of the lidar ranging in a dual-sensor architecture. For evaluation purposes, we assume perfect knowledge of $\theta$ in this paper.

Fig. 8

Classification accuracy of $f_r$, DoLP, and hybrid features with the SVM classifier using fivefold cross-validation against the material database, showing (a) ${\theta }_1$ varied from 0 deg to 70 deg with no noise added and (b) Gaussian noise added to observations at 70 deg such that SNR is varied from 3 to 10 dB.

4.5.

Discussion of Results

The classification accuracy of all scenarios evaluated on our dataset is greater than 20%. With the 19 classes considered, a completely random guess would result in less than 5.3% chance of correct classification. The performance is enabled by having a known observation angle. When considering a single known observation angle, results of $f_r$ and DoLP are very similar [Fig. 8(b)]. However, when combinations of angles are considered (Tables 3 and 4) $f_r$ consistently outperforms DoLP. In fact, the more angles utilized, the better the performance. This is because the actual measurements (shown in Figs. 6 and 7) of most materials have signatures that vary with observation angle. In all scenarios combining the features in a hybrid architecture significantly improves performance. As previously mentioned, a standalone passive polarimeter is not capable of obtaining observation angle without lidar point-cloud information. Therefore, utilizing only the DoLP feature would still require a lidar system. We believe 6 dB is a reasonable evaluation point for SNR, based on our experience with lidar and infrared imaging systems. At 6 dB, the proposed technique achieves 91.1% material classification accuracy using SVM.

When comparing classifier techniques (Table 3), SVM obtains the best results. This could be due to the limited parameter space we explored with each classifier (shown in Table 1). Optimizing these parameters for the specific dataset could improve classification accuracy of each method. We also notice that some SVM classifiers require on the order of 10 times longer to train than other classifier types (but performance metrics on training time are not presented because the metric is highly dependent on computational hardware). We recommend that the type of classifier utilized in future work should be carefully selected for each individual application (by considering the amount of training data, dimensionality of the data, training time, number of the features, number of classes, and class separation).