We investigate the augmentation of active imaging with passive polarimetric imaging for material classification. Experiments are conducted to obtain a multimodal dataset of lidar reflectivity and polarimetric thermal self-emission measurements against a diverse set of material types. Using the assumption that active lidar imaging can provide high-resolution three-dimensional spatial information, a known surface orientation is utilized to enable higher fidelity classification. Machine learning is applied to the dataset of monostatic lidar unidirectional reflectivity and passive longwave infrared degree of linear polarization features for material classification. The hybrid sensor technique can classify materials with 91.1% accuracy even with measurement noise resulting in a signal-to-noise ratio of only 6 dB. The application of the proposed technique is applicable for the classification of hidden objects or could assist existing spatial-based object classification.