As optical distribution frames (ODFs) carry increasing amounts of data, issues relating to their port management are becoming crucial. Image backhaul inspection has been widely accepted as an effective technology to monitor the occupancy status of ODF ports. However, the massive amount of on-site images requires a large team of off-site technologists to manually identify the occupancy status of each ODF port, which is time-consuming. We employ the you only look once version 3 (YOLOv3) network to automatically recognize ODF port occupancy. The YOLOv3 is a state-of-the-art convolutional neural network that has been shown to be very efficient for object detection in terms of processing speed and accuracy for the common objects in context. To accommodate ODF images with densely arranged small objects, high resolutions, closely spaced adjacent ports, and occlusion, we modified the original YOLOv3 with four-scale feature fusion, anchor box dimension clustering, and soft nonmaximum suppression filtering. Experiments showed a 7.38% increase in the original YOLOv3 detection accuracy rate of 91.45%. The new method can update the image backhaul inspection to automatically realize port resource management. The number of required port management technologists is considerably reduced, and the accuracy of port resources is increased, resulting in significant network investment savings.