Spectral and spatial details abound in hyperspectral data. However, there are few labeled samples available, and obtaining them is expensive and time-consuming. As a result, improving the classification accuracy of HSI with limited training samples is a critical challenge. We propose a double-branch network with attention feature fusion (DBN-AFF) for HSI classification. One branch extracts spectral features, and the other extracts spatial features. It is capable of effectively preventing cross-branch interference. Furthermore, the AFF module is used instead of the channel concatenation method. It effectively fuses spectral and spatial features by considering the relationship between different channels. On the Indian Pines, Pavia University, and Salinas Valley datasets, DBN-AFF achieves 97.44%, 98.37%, and 97.60% on limited training samples, respectively.