You only look once version 5 (YOLOv5) algorithm uses one convolutional layer for both classification and localization tasks. They share the same set of weights, leading to interference between the two tasks. Thus, the application of this method in remote sensing scenario is very limited. To solve the problem, we decouple the detection head of the YOLOv5 algorithm so that these two tasks can be separated. Furthermore, we achieve high quality detection in remote sensing scene with task refinement. Specifically, we propose a weight allocation method based on Huffman coding theory to ease imbalance of the number of categories and improve the classification accuracy. If the number of a category is large, a small weight is set. Otherwise, a large weight is given. To improve the localization accuracy, we use the efficient intersection over union (IOU) loss to alleviate the problem of penalty failure caused by central IOU loss. In this way, the predicted boxes are closer to the ground truth. A large number of experiments were carried out on DOTA and UCAS-AOD datasets to verify the effectiveness of our proposed method. Compared with baseline, the mean average precision totally increased by 2.29% in DOTAv1.5 dataset. Simultaneously, our method also showed superior performance when comparing with start of the art algorithms.