Defect detection based on deep learning has been applied in many fields and has made great progress in recent years. However, the problems of high false detection rate in complex scenes and weak multiscale detection capability have not been solved well. To solve the above problems, a damper defect detection network (DDDNet) is proposed, which includes a cognitive preprocessing network (CogniPrepNet) and a heterogeneous feature pyramid network (HeteroFPN). CogniPrepNet enhances targets and mitigates the impact of complex backgrounds through the ghost-shuffle module. HeteroFPN obtains feature maps with a wider range of scales through a many-to-one mapping relationship to achieve high-quality multiscale feature fusion. Moreover, a fast proportional intersection over union is designed to improve the sensitivity of geometric factors during the bounding box regression. The experimental results on damper-DET dataset show that the DDDNet can achieve 98.25% mAP and 34 FPS detection speed in complex background and multiscale target scene.