Accurate semantic segmentation of images has long been a research priority in remote sensing. However, the presence of geometrically complex and spatially diverse objects increases the difficulty in simultaneously obtaining coherent and accurate labeling result. To solve this challenge, our study combined multiscale geometric feature extraction with convolutional neural network and proposed a new U-shaped contourlet network (USCNet) for segmentation from high-spatial-resolution remote sensing images (HSRRSIs). This network is designed to learn and characterize the geometric features present in HSRRSIs. The USCNet first transforms the original dataset into a pyramidal structure containing multiscale and multidirectional geometric information and then fuses the spatial and geometric features to extract high-level semantic information. This network has two advantages: (1) coarse-to-fine spatial features are learned efficiently using a hierarchical learning structure and (2) the multiscale learning scheme captures geometric information in different directions. The results of extensive experiments conducted on two remote sensing datasets (the International Society for Photogrammetry and Remote Sensing Vaihingen and Potsdam challenge datasets) show that the proposed approach outperforms several state-of-the-art semantic segmentation methods.