YOLOv5 is a one-stage detector that achieves appealing performance in real-time object detection. When it comes to remote sensing object detection, there are many small objects in the scene, and the number of objects in different categories varies significantly. Directly applying YOLOv5 for remote sensing object detection usually ignores these small objects. Furthermore, imbalance among different categories also makes the model prone to some majority categories. In this way, we propose a smallness and imbalance-aware head and apply it to YOLOv5. The improved model is named SIA-YOLOv5. To be specific, a normalized Gaussian Wasserstein distance is designed to replace the commonly used intersection over union in the regression process, which substantially improves the localization accuracy for small objects. Meanwhile, an adaptive weighting strategy is designed to make a flexible emphasis on the classification accuracy among different categories, which relieves the unstable performance caused by imbalanced data. In addition, BiFPN and coordinate attention are utilized for better feature extraction. Experimental data and analysis have demonstrated the effectiveness of the proposed method.