Significance: The Mueller matrix decomposition method is widely used for the analysis of biological samples. However, its presumed sequential appearance of the basic optical effects (e.g., dichroism, retardance, and depolarization) limits its accuracy and application.

Aim: An approach is proposed for detecting and classifying human melanoma and non-melanoma skin cancer lesions based on the characteristics of the Mueller matrix elements and a random forest (RF) algorithm.

Approach: In the proposal technique, 669 data points corresponding to the 16 elements of the Mueller matrices obtained from 32 tissue samples with squamous cell carcinoma (SCC), basal cell carcinoma (BCC), melanoma, and normal features are input into an RF classifier as predictors.

Results: The results show that the proposed model yields an average precision of 93%. Furthermore, the classification results show that for biological tissues, the circular polarization properties (i.e., elements m₄₄, m₃₄, m₂₄, and m₁₄ of the Mueller matrix) dominate the linear polarization properties (i.e., elements m₁₃, m₃₁, m₂₂, and m₄₁ of the Mueller matrix) in determining the classification outcome of the trained classifier.

Conclusions: Overall, our study provides a simple, accurate, and cost-effective solution for developing a technique for classification and diagnosis of human skin cancer.