2.1.

Progress in Ruminant Identification Technology

Borchers et al.⁸ conducted a comprehensive work on rumination behavior in cattle, employing six different tri-axial accelerometer technologies. These included the cattle manager sensor, used to monitor rumination and feeding time, and the intelligent bow sensor, used to track cattle and monitor their rumination behavior and time spent in feeding areas. Visual observation was utilized to generate behavior time lengths, recording the start and end times of rumination behaviors occurring within a day. In another work, Bishop-Hurley et al.⁹ employed a collar system equipped with tri-axial accelerometers and magnetometers to investigate feed additives’ impact on cattle feeding behavior. In addition, they analyzed and developed behavior models using multivariate time series data to address potential issues with the sensors. Arablouei et al.¹⁰ developed a specialized pipeline for pre-processing, feature extraction, and cattle behavior classification using measurement data. This pipeline was designed to address resource constraints. They collected data from 10 cattle using tri-axial accelerometer sensors on collar tags to record their rumination and other behaviors. Watt et al.¹¹ utilized acoustic methods to measure rumination activity in cows and discovered a positive correlation between dry matter intake and rumination time. Furthermore, they conducted a 10-day baseline detection for frequency analysis of rumination time.

Rombach et al.¹² validated the effectiveness of the ruminant welfare system (RWS) in measuring cows’ rumination behavior during barn grazing and supplementation periods. They improved the algorithm used in the RWS system. They found that the two converters used in the experiment were better at distinguishing rumination and feeding behavior in cows during other activities. Braun et al.¹³ established feature pressure curves by comparing data obtained from pressure sensors with 24-h direct observation data, demonstrating the regularity of chewing during rumination and generating a uniformly regular waveform. The consistency of the results was confirmed by comparing the data obtained through direct observation and pressure sensors. Gregorini et al.¹⁴ conducted a 21-day strip grazing experiment on a perennial ryegrass pasture with 8 cows equipped with high resolution (HR) tags and rumination collars to record rumination behavior and chewing activity during rumination. They calculated and registered rumination time based on the mean interval time of chewing actions, ultimately verifying the cows’ daily rumination pattern under the constraints of the pasture’s time. Handcock et al.¹⁵ demonstrated the potential of a wireless sensor network to monitor cattle behavior by combining global positioning system collars with satellite imagery, providing high temporal resolution. Schirmann et al.¹⁶ evaluated changes in rumination and feeding behavior before and after calving using time standards. Zehner et al.¹⁷ developed a novel scientific detection device to automatically measure stable-fed cow rumination and feeding behavior using a universal algorithm based on animal-specific learning data, with two software versions for the system. Pereira et al.¹⁸ evaluated the correlation and differences between direct visual observation and sensor data analysis by recording data for 6 h using a trained observer and ear-tag accelerometer sensors. Ruuska et al.¹⁹ proposed a pressure sensor system to measure rumination time in cattle and validated its effectiveness through comparative experiments.

Porto et al.²⁰ proposed a computer vision-based automated detection system for free-stall resting behavior in cattle that utilizes a Viola–Jones algorithm for cow lying behavior detection. The system captures image data of the research area through a multi-camera recording system and validates the system’s effectiveness by comparing the detection results with visual recognition results. Yazdanbakhsh et al.²¹ developed an intelligent system for continuous monitoring of the health status of each animal in livestock using sensors installed on animals. The experiment showed that higher sensitivity and specificity could be achieved using an integrated classifier in the wavelet domain. Arcidiacono et al.²² proposed a model based on an acceleration thresholding algorithm to detect cattle behavior by predicting the threshold of behavior onset from data obtained under specific experiments in the barn where the cattle is housed, thus determining the behavior of the cattle. Viazzi et al.²³ developed a movement model based on lame cow walking behavior, increasing both single threshold accuracy and actual positive rate by representing the behavioral features hierarchically, effectively improving the accuracy of rapid and precise identification of lame cow walking on the farm. Pahl et al.²⁴ analyzed feeding characteristics recorded by weighing troughs and rumination time changes recorded by acoustic sensors based on the feeding characteristics before and after cow estrus. Lee et al.²⁵ organized and analyzed the predicted physiological parameters and various data from a wearable wireless sensor system for cows. The sensors used in the system primarily focus on rumination behavior, including ear tags, collars, and reticulum boluses. Jonsson et al.²⁶ introduced individual animal lying balance by combining information from step sensors and leg tilt sensors, deriving a new change detection scheme, and establishing a change detection algorithm to describe cow behavior state within a given time interval using a binary variable. Braun et al.²⁷ employed an automated system to assess feeding and rumination variables in a cohort of 300 dairy cows over 24 h. The method utilized pressure sensors integrated into neck collars to capture jaw movements. Subsequently, it analyzed vital parameters such as the duration of rumination, number of chewing cycles, number of chews per cycle, and units of chews per cycle. These studies have contributed significantly to understanding cattle rumination behavior and have used various sensor technologies and data analysis techniques to achieve meaningful insights.

2.2.

Advances in Deep Learning-Based Cattle Regurgitation Recognition Technology

Jiang et al.²⁸ proposed implementing the fast layered YOLO version 3 (FLYOLOv3) network for detecting essential parts of cattle in complex scenes based on the filter layer. They integrated a customized filter layer with an average filtering algorithm and leaky rectified linear unit (leaky ReLU) function to reduce training interference. Shakeel et al.²⁹ introduced an innovative behavior recognition and computing scheme to predict cattle behavior. The proposed method used a deep recurrent learning paradigm to cycle the recognition pattern and classify abnormal situations based on differentiated data patterns. Chen et al.³⁰ evaluated the latest developments in computer vision methods for recognizing cattle behavior based on animal productivity, health, and welfare. They analyzed the effect of image segmentation, recognition, and behavior recognition using traditional computer vision and deep learning methods, listing the progress of crucial research in this field. Peng et al.³¹ combined long short-term memory (LSTM) networks to detect and identify cattle behavior using inertial measurement units and classify behaviors such as regurgitation and feeding. LSTM - recurrent neural network (LSTM-RNN) model training showed potential for development in cattle regurgitation detection. Tamura et al.³² investigated the correlation between cattle behavior and acceleration data collected using a tri-axis neck-mounted accelerometer. They proposed the feasibility of improving behavior classification accuracy through machine learning. By recording the characteristic acceleration waves of eating, rumination, and lying down during visual observation of cattle behavior, they combined the farm dataset for decision-tree learning and ultimately verified the accuracy of decision-tree knowledge.

Fuentes et al.³³ proposed a hierarchical cattle behavior recognition method based on deep learning, incorporating spatiotemporal information. The framework includes appearance features at the frame level and spatiotemporal information containing more contextual time features. The designed system detected and located rumination behavior in multiple cattle in video frame regions. The method was validated using datasets captured in both day and night environments, showing effective recognition of 15 types of hierarchical activities. McDonagh et al.³⁴ continuously monitored 46 cattle and used image recognition technology to predict their behaviors. Non-local neural networks were trained and validated on video clips for each behavior, showing successful recognition and classification of behaviors over extended periods. Dutta et al.³⁵ classified cattle behavior recorded by a collar system equipped with a tri-axial accelerometer and magnetometer using machine learning techniques. A hybrid framework was developed to work the natural structure of sensor data, and various classification methods were compared to verify the model’s superiority in representing rumination behavior. Chen et al.³⁶ proposed an intelligent monitoring method for cattle rumination behavior based on video analysis. They used the MEAN-SHIFT algorithm to track the movement of the cattle’s lower jaw and extract the centroid trajectory curve of the cattle’s mouth movement from the video, enabling monitoring of cattle rumination behavior. Li et al.³⁷ proposed a multi-target monitoring method based on optical flow and frame difference for cattle rumination behavior. Using optical flow and frame difference, they identified candidate rumination cattle oral regions and tracked the cattle’s mouth within the area, enabling automatic monitoring of the rumination cattle’s mouth area. Tamura et al.³⁸ developed a method for detecting chewing speed using a tri-axial accelerometer connected to the cattle’s neck. Using the fast Fourier transform algorithm, they analyzed neck vibration as rumination movement and calculated chewing time. Dutta et al.³⁹ proposed an intelligent IoT device for cattle monitoring to identify rumination behavior and collect rumination data through classification. The device was mounted on the cattle’s neck, and data were transmitted to IoT servers and a cellular global-local space modulation module. Xu et al.⁴⁰ constructed an advanced instance segmentation framework MASK-RCNN to address the problem of cattle dataset occlusion and overlap in complex scenes. They compared classical algorithms and validated the best threshold and complete detection for cattle herd data in different situations, demonstrating the effectiveness of the proposed framework. These studies provide valuable insights into monitoring cattle behavior and have practical applications in the agricultural industry.^41–43

2.3.

Summary

The present work introduces an innovative technique for bovine rumination identification that addresses various perspectives. While rumination detection techniques based on wearable devices have limitations and may not meet animal welfare requirements, methods using pressure sensors and tri-axial accelerometers to detect rumination tend for prioritizing data collection over accurate identification. In addition, some approaches rely on identifying bovine body or head regions, which may need more accuracy in rumination recognition. In contrast, our proposed method focuses on bovine rumination identification by extracting features from the upper and lower jawbone skeleton. Specifically, we utilize skeleton extraction technology to extract the upper and lower jawbone skeleton features from the bovine mouth region. These features are then processed using the UD-YOLOv5s network architecture, enhancing the identification process’s accuracy and efficiency. We optimize the loss function to improve performance. By adopting this method, we achieve more accurate and efficient bovine rumination identification compared to traditional techniques. It ensures higher precision and considers the animals’ welfare, making it a promising step toward effective and humane bovine rumination identification.

3.1.

UD-YOLOv5s System Architecture

This paper introduces UD-YOLOv5s for accurately identifying cattle grazing behavior. The proposed approach leverages the YOLOv5s network to extract crucial skeletal features from upper and lower jawbones. We combine the SHD and KF algorithms to achieve this and process the original feature map. Subsequently, the focus block is utilized for image segmentation, and the output is unsampled and fed into the convolution layer. The feature fusion module combines these features, subsequently connected to the pre-processing layer. Finally, we optimize the output using the GIoU loss function. The overall architecture of our system is shown in Fig. 1. The ruminant behavior of cattle primarily manifests through the skeletal movements of their upper and lower jaw regions. In this paper, we extract the skeleton features of the upper and lower jaw regions separately to discriminate ruminant behavior accurately. Specifically, the upper jaw region exhibits linear movement during cow rumination. We employ the SHD algorithm to extract the upper jaw skeleton structure from the cow’s image. The SHD algorithm is proficient in extracting the primary structural lines of objects by refining the edges of the image, thereby obtaining the skeleton features of the upper jaw. For the image of the cow’s upper jaw, the SHD algorithm effectively simplifies the complex edge structure into skeleton lines, thereby extracting the main structural features of the upper jaw. In addition, we utilize the KF, a filtering algorithm for state estimation while extracting the cow’s jaw skeleton features. The KF algorithm estimates the state of the jaw based on observed jaw position and motion trajectory, subsequently extracting the jaw’s motion features and skeleton information. The upper and lower jaw skeleton feature extraction captures distinct aspects of the cow’s ruminant behavior. Precisely, the upper jaw skeleton feature reflects the morphological characteristics of the cow’s mouth while the lower jaw skeleton feature reveals the movement characteristics of the cow’s lower jaw. A more comprehensive and integrated representation of ruminant behavioral features is achieved by fusing these two features. As a target detection network, the YOLOv5s network can accept multi-channel inputs. Therefore, the upper and lower jaw skeleton features can be effectively utilized as network inputs to achieve a comprehensive feature representation and analysis. This enhances the accuracy and effectiveness of our method for accurately identifying cattle grazing behavior.

Fig. 1

System architecture diagram.

3.2.

UD-YOLOv5s Method Design

3.2.1.

Upper and lower jaw skeleton feature extraction

Skeleton extraction is a valuable technique to detect essential points within an image and connect them sequentially, forming skeleton information. In the specific case of the cattle mouth region, the skeleton can be divided into two parts: the upper jaw and the lower jaw skeletons. The area of interest is first segmented into the upper and lower jaw regions to label the feature points in the original image. The feature points in the upper jaw are denoted as U1 to U6, corresponding to their coordinate positions starting from the left of the mouth area. Similarly, the feature points in the lower jaw are labeled as D1 to D5. Each pair of feature points between the upper and lower jaw is represented by a unit vector, such as U1→D1, connecting the respective points. However, due to the absence of a corresponding feature point for U6 in the lower jaw region, it is excluded from the skeleton. Figure 2 visually shows the labeled vital points in the cattle’s mouth region, offering an overview of the identified feature points.

Fig. 2

Point marker map of the bull’s mouth region.

In this paper, we have employed the SHD detector, a highly accurate tool with a 90.90% success rate in identifying crucial points on the head, showcasing exceptional detection precision. To tackle the challenges posed by demanding conditions such as occlusion caused by cattle enclosures, complex environments, and overlapping bodies, we have innovatively utilized an 8-layer stacked hourglass network (8SHD). This network effectively transforms the original image into a feature map, creating a robust skeletal extraction model. The architecture of this model is represented in Fig. 3.

Fig. 3

SHD model diagram.

This paper presents an advanced algorithm designed to extract crucial features of the mandible skeleton in cattle, which is vital in complex motion patterns. The method involves a multi-step process to ensure accuracy and reliability. First, the algorithm filters out low-confidence vital points, ensuring that only the most reliable maxillary structure critical points are considered. This selection is based on whether the output heat map exceeds a predefined threshold. An elliptical curve is employed to approximate the mandibular region, which aids in accurately capturing its shape. The algorithm then proceeds to identify vital points based on specific characteristics of the mandibular skeleton. These points serve as the foundation for further analysis. Next, a predictive algorithm is utilized to set key point features, refining the representation of essential points in the mandible. The algorithm considers the dynamic nature of the mandibular motion sequence, treating it as a dynamic system. During the opening and closing of the mandible, the left mouth edge point is selected as the original feature point, acting as a reference for the entire motion sequence. The curve formation is thoughtfully arranged by incorporating the mandibular skeleton feature and setting the adjacent bone distance to 0.1 cm per unit distance. In addition, the neighboring nodes of each unit period are considered adjacent skeleton feature points. By employing this sophisticated approach, the algorithm accurately models the motion pattern of the mandible.

When $\mathrm{AP}=1$ is recorded as the initial frame $K$, at this time, the jaw joint point $B_k$ gets the anchor frame and sits marked as $(x_1,y_1)$. Currently, the next frame joint point $B_{k|k+1}$ position can be predicted by the KF algorithm as

Eq. (1)

$$B_{k|k+1}=A(x_1,y_1),$$

where $A$ denotes the state transfer matrix, and the site-tag of the next frame $B_{k+1}$ is predicted to be $(x_2,y_2)$. The covariance matrix is calculated according to $B_k$ and $B_{k+1}$ prediction variable densities as

Eq. (2)

$$P_{k|k+1}=A{P}_{k+1|k+2}{A}^T+Q,$$

where $P_{k|k+1}$ is denoted as the covariance matrix of $B_k$, $P_{k+1|k+2}$ is represented as the covariance matrix of $B_{k|k+1}$, $Q$ is the process noise, and $T$ is the time stamp. The position of the detected node after processing is determined with the predicted position, and the updated state is calculated by Kalman gain. The coordinates of the matching node are replaced with the predicted coordinates. The gain is calculated as

Eq. (3)

$$K_k=P_{k|k-1}{H}^T{(H{P}_{k|k-1}{H}^T+V)}^{-1}.$$

The replacement state after gain is calculated as

Eq. (4)

$$W_{k|k}=W_{k|k-1}+K_k(X_k-H{x}_{k|k-1}).$$

The error covariance is shown as

Eq. (5)

$$P_{k|k}=(1-K_{k}H)P_{k-1|k-1},$$

where $K_k$ denotes the Kalman gain of frame $k$; $W_k$ denotes the updated lower jaw position; $P_k$ denotes the covariance matrix. We denote the obtained initial key point of the upper jaw as p1, the next predicted node of the frame as p2, and the adjacent algorithm-predicted nodes as p3 and p4. The initial key point of the lower jaw as p5, the next predicted vital point of the frame by the algorithm as p6, and the adjacent nodes as p7 and p8. The skeleton critical point representation is shown in Fig. 4.

Fig. 4

Cattle skeleton key point representation diagram: (a) original image, (b) key point extraction, and (c) key point of the jaw.

Information setting based on the jaw key point representation. Key point parameters are shown in Table 1.

Table 1

Key point information based on algorithm settings.

Key point representation	Key points description
P1	Initial trace box key points
P2	Next frame forecast point
P3	Next frame forecast point
P4	Next frame forecast point
P5	Initial trace box key points
P6	Next frame forecast point
P7	Next frame forecast point
P8	Next frame forecast point

3.2.2.

Design of UD-YOLOv5s network

The YOLOv5s object detection algorithm is renowned for its exceptional detection accuracy and lightweight architecture. In this research, we introduce a groundbreaking approach called the UD-YOLOv5s network, specifically designed for detecting the mouth area of cattle and recognizing their rumination behavior. To achieve this goal, our methodology begins by resizing the input image to $640\mathrm{px}\times 640\mathrm{px}$. Subsequently, we employ a slice operation to divide and connect the image, generating a processed image for further analysis. We incorporate spatial pyramid pooling and the convolutional block attention module (CBAM) to enhance the receptive field and feature expression. These components capture essential information from various spatial scales, optimizing the network’s performance. The data are then passed to the convolutional layer, where a bottleneck mechanism is employed for efficient feature extraction, thereby reducing the computational burden. The subsequent step involves feeding the up-sampling connected convolutional layer into the feature fusion module. This integration is executed bottom-up, employing the ReLU activation function, up-sampling connection, and GIoU loss function for effective optimization. Finally, the output of the feature map includes target bounding boxes, class probabilities, and confidence scores. A decoder completes this process, facilitating the precise identification and localization of cattle mouth regions. The architecture of the UD-YOLOv5s network is visually shown in Fig. 5, providing an overview of our proposed model’s intricate yet powerful design.

Fig. 5

UD-YOLOv5s network structure diagram.

In the context of bull’s mouth image analysis, addressing potential node mismatches that may occur when capturing images using a camera is essential. To ensure the accuracy and reliability of the skeleton features, we employ the object key point similarity (OKS) as a metric for evaluation, as shown in Eq. (6). OKS is a crucial measure to assess the similarity between the detected critical points in the image and the ground truth positions of these key points. Using OKS as an evaluation method, we can quantitatively gauge the alignment between the skeleton features extracted from the image and the expected positions of these features. The precise definition and calculation of OKS are shown in Eq. (6), allowing us to effectively evaluate and validate the correctness of the identified skeleton features compared to the ground truth. By incorporating the OKS evaluation method, we can enhance the robustness of our bull’s mouth image analysis, ensuring that the skeleton features accurately represent the underlying anatomical structures even in scenarios where node mismatch may occur during image capture

Eq. (6)

$$O_{ks}=\frac{\sum _i{e}^{-\frac{di}{2Sab}}}{\sum b},$$

where $O_{ks}$ denotes the similarity of the key points of the maxillary skeleton, $a$, $b$ represents the visibility of the critical points, $S$ the pixels of the individual feature points of the skeleton, $d_i$ the Euclidean distance between the actual coordinates of the critical points and the predicted coordinates, and $i$ the normalization factor of the key points. After inputting the regurgitated test set into the model, the average precision (AP) is calculated by calculating the OKS ratio to the skeleton individual node threshold, and determining whether the predicted target detection node coordinates match the actual coordinate points, if $\mathrm{AP}=0$, the current image detection error is determined and the next image is input; $\mathrm{AP}=1$, the skeleton node anchor frame is generated by the adaptive anchor frame mechanism, and the results are output to the backbone layer.

The backbone network layer and neck network layer are our model’s core components and have been optimized using a lightweight approach. To enhance detection accuracy, we have incorporated the CBAM into the backbone layer for the skeleton nodes. CBAM comprises two main modules: the channel attention (CHN) module and the spatial attention (SPA) module. By leveraging CHN, the model can focus on relevant features within each channel, effectively improving the overall representation. Subsequently, the feature map is passed through the SPA, enabling the model to extract crucial spatial information, further enhancing feature extraction. For a visual representation of the CBAM attention mechanism, refer to Fig. 6.

Fig. 6

CBAM attention mechanism structure.

In CHN module, the original feature map $X$, which is the product of height $H$, width $W$, and number of channels $N$, is pooled to obtain the channel map, and then processed by multilayer perceptron (MLP) to obtain the feature weights, combined with the ReLU activation function to obtain the channel weight coefficients $M_n$. The product of $M_n$ and $M$ is used as the scaled channel feature map $Y$, and $M_n$ is calculated as

Eq. (7)

$$M_n=\partial (\mathrm{MLP}(\text{Pool}(X))),$$

where $\partial$ represents the ReLU activation function, MLP the multilayer perceptron, and Pool the adaptive pooling operation. In the SPA module, the new feature map $Y$ is pooled to obtain the channel map for stitching, and the spatial weight coefficient $M_s$ is obtained after $3*3$ layer convolution and ReLU activation function, and the product of $M_s$ and the feature map $Y$ is used as the output feature map $Z$. The formula of $H$ is as in Eq. (8)

Eq. (8)

$$M_s={\theta }^{3*3}(\text{Pool}(Y)),$$

where ${\theta }^{3*3}$ means the $3*3$-layer convolution and $Y$ the channel feature map. The filtered feature map $Z$ is input to the Neck layer, as shown in Fig. 7.

Fig. 7

UD-YOLOv5s feature extraction map.

The neck layer employs a bottom-up path aggregation feature pyramid to transfer feature information efficiently. This pyramid facilitates the integration of feature maps corresponding to key points’ information. The process involves convolving the feature map of each skeleton key point with a step size of 1, which effectively fuses the features. Up-sampling techniques are utilized to accomplish the feature fusion, and the convolution module consists of a series of $x$ residual structures resource units. This integration process enhances the overall representation and prepares the feature information for the subsequent detector. Ultimately, the prepared feature information is fed into the detector, where regurgitated behavior recognition occurs, as illustrated in Eq. (9)

Eq. (9)

$$V=\{\begin{array}{l}\text{Pool}(Z^i{)}^{1*1}\to \text{Upsample}\\ \text{Prediction}({\mathrm{Mod}}^x)\end{array},$$

where $V$ represents the feature information, Upsample the up-sampling operation, ${\mathrm{Mod}}^x$ the convolution module integrated by $x$ residual structures, and Prediction the detector prediction operation. The neck layer architecture and output are shown in Fig. 8.

Fig. 8

UD-YOLOv5s path aggregation neck network architecture diagram.

The output tensor is partitioned into multiple small tensors, each corresponding to a network cell and its surrounding prediction frame. Within each small tensor, computations are performed to determine the target confidence, category probability, and bounding box coordinates. The final prediction takes the form of a three-dimensional tensor, where the first dimension represents the identification of each grid cell, the second dimension means the identification of each predefined frame, and the third dimension contains information related to target confidence, category probability, and bounding box coordinates. The upper and lower jaw skeleton feature extraction method is employed to obtain multiple anchor frames in the mouth region. This technique can derive anchor frames based on the skeleton features of the upper and lower jaws. Specifically, the center point of the upper jaw anchor frame serves as the base point, and it is connected to the center point of the lower jaw anchor frame. The Euclidean distance between these two anchor points is then calculated as

Eq. (10)

$$d=\sqrt{{(x_1-x_2)}^2+{(y_1-y_2)}^2},$$

where $x_1$, $x_2$, $y_1$, and $y_2$ denote the horizontal coordinates of the center point of the upper and lower jaw anchor frames and the vertical coordinates of the center point of the upper and lower jaw anchor frames, respectively. Take the minimum value of $\mathrm{Min}(d^K)$, at this time $k$ frame is recorded as a chewing determination, regurgitation is consistent with chewing periodicity, so the determination of the number of regurgitation $\alpha$ as shown in Eq. (11)

Eq. (11)

$$\alpha =l/({\mathrm{Min}}^k-{\mathrm{Max}}^{k+1}),$$

where ${\mathrm{Min}}^k$ denotes the minima on the mastication curve in frame $k$, ${\mathrm{Max}}^{k+1}$ the maxima in the adjacent next frame, and $l$ the mastication curve length. The final recognition prediction output is completed in the prediction layer.

3.3.

Optimization of UD-YOLOv5s

Recognizing ruminant behavior poses a significant challenge due to an inherent dataset imbalance, with a limited number of ruminant samples compared to an abundance of non-ruminant instances. Traditional YOLOv5s rely on the IoU loss function, which needs to be more accurate in capturing the overlap between the target and authentic frames. This paper tackles this issue and aims to enhance model learning by introducing the GIoU loss function, offering a more precise measurement of frame overlap. We note that we retained all data in our work to maintain dataset integrity.

4.1.

Data Acquisition

The experiment was conducted at a ranch in Yuan Yang, Henan Province, China, involving 120 mature Simmental cattle. To ensure comprehensive coverage of cattle activities, a camera was placed at a 45 deg angle above the ranch. Ranch caretakers meticulously screened the video data, and instances of cattle ruminating within 3 h of feeding at 5 AM and 5 PM were selected. Cattle were observed to be resting quietly during other periods. Subsequently, a frame-by-frame analysis was performed to process the video frames into image datasets. These resulting data files were accurately labeled based on their contents, with a resolution of $640\mathrm{px}*480\mathrm{px}$. Both training and testing utilized the captured cattle behaviors. In addition, different degrees of occlusion were incorporated into the data collection process to reflect real-life farm conditions. The definition of cattle rumination is shown in Table 3, ensuring transparency and replicability of the work.⁴⁴

Table 3

Definition of cattle ruminant labeling.

4.2.

Data Pre-Processing

In this work, the images utilized were impacted by low quality, mainly due to overlapping shapes caused by obstructions from the farm cattle pen, cattle movement, and poor adaptability to varying lighting conditions. To ensure data integrity, these low-quality images were excluded from the dataset. To enhance the generalization ability of the recognition network, data augmentation techniques were employed. Rotation and the mosaic data augmentation technique effectively augment the dataset. In addition, classic data augmentation techniques, such as image flipping, scaling, cropping, translation, noise addition, and contrast modification, were applied to diversify the dataset further. For a visual representation of the augmented dataset, refer to Fig. 9. These augmentation approaches collectively improve the model’s ability to recognize cattle behaviors amidst challenging environmental conditions and variations in image quality.

Fig. 9

Cattle expansion data chart: (a1)–(c1) original image, (a2) image denoising, (b2) image cropping, and (c2) image scaling.

Mosaic enhancement is a widely adopted image processing technique in object detection. In the YOLOv5s network, the cut and mix data augmentation algorithm is employed, effectively expanding the dataset and enhancing its diversity.^45,46 Similarly, in UD-YOLOv5s, the stitcher method is incorporated as a data augmentation approach to tackle the issue of low accuracy in object detection caused by poor image quality resulting from various environmental factors during image capture.⁴⁷ Figure 10 visually shows the process diagram for the mosaic enhancement technique employed in UD-YOLOv5s. By leveraging the mosaic enhancement and stitcher methods, the model’s ability to detect and recognize objects, even in challenging conditions with compromised image quality, is substantially improved. These techniques are pivotal in optimizing the performance of UD-YOLOv5s for object detection tasks.

Fig. 10

Mosaic data enhancement flowchart for UD-YOLOv5s.

The mosaic-enhanced image of the data in this paper is shown in Fig. 11.

Fig. 11

Mosaic image enhancement map: (a1)–(c1) original image, (a2) brightness adjustment, (b2) contrast adjustment, and (c2) saturation adjustment.

In this context, consider Figs. 11(a1)–11(c1) as the original images and Figs. 11(a2)–11(c2) as the processed images. For data annotation, we utilized the labeling tool, and the resulting dataset is named the “cattle dataset.” This dataset comprises two distinct categories: “ruminate” and “other,” it encompasses a total of 1500 processed datasets. The data annotation details can be observed in Fig. 12. The use of the labeling tool ensures accurate and consistent annotation. At the same time, the division of the dataset into “ruminate” and “other” categories facilitates targeted analysis of cattle behavior. This comprehensive dataset lays the foundation for robust training and evaluation of the model, enabling effective detection and recognition of cattle activities.

Fig. 12

Data annotation chart.

4.3.

Experimental Environment and Parameter Settings

The experiments were conducted on a LINUX operating system, utilizing a robust hardware setup consisting of 16 GB of RAM, an NVIDIA GEFORCE RTX3070 graphics card for GPU acceleration, and a Core i7 CPU, along with an Intel(R) Core (TM) i7-10750H CPU@2.60 GHz 2.59 GHz processor for network training configuration. The model was built using the PyTorch deep learning framework, known for its efficiency and flexibility. Careful attention was given to the hyperparameter settings to ensure optimal performance during the training phase, as outlined in Table 4. These hyperparameter configurations play a crucial role in shaping the model’s learning process, enabling it to capture intricate patterns and features in the cattle dataset effectively. The robust hardware setup and well-tuned hyperparameters contribute significantly to the model’s accuracy and efficiency, enabling comprehensive training and evaluation of the object detection model for cattle behavior recognition.

Table 4

Hyperparameter setting.

To thoroughly evaluate the efficacy of the YOLOv5s network in detecting the upper and lower jaw regions of cattle during rumination behavior, we conducted an extensive rumination recognition detection experiment using the cattle dataset introduced in this paper. To ensure a fair and comprehensive comparison, we carefully selected MEAN-SHIFT, MASK-RCNN, and YOLOv3 as the control group networks, considering their similarities, versatility, and advancements in the field.

MEAN-SHIFT, a clustering algorithm, segmented images by analyzing pixel color distribution. We extracted mouth features from video sequences and performed clustering operations using MEAN-SHIFT to recognize cattle rumination behavior occurring in the mouth area.

MASK-RCNN, an enhanced algorithm capable of simultaneous object detection and instance segmentation, played a vital role in the experiment. We labeled the cattle rumination dataset and utilized pre-trained models to detect and locate the mouth area of the cattle. The region information was then fed into the MASK-RCNN network to perform mask segmentation, generating a segmentation mask for the cattle’s mouth. Subsequently, we extracted the feature vector and performed classification to identify cattle rumination behavior.

The YOLOv3 algorithm involved video processing software to extract images containing cattle rumination behavior and conducted object detection to obtain the position information of the cattle’s mouth. We pulled relevant feature information, such as motion trajectory and color, and employed statistical analysis to determine the presence of rumination behavior.

To ensure a fair comparison, we maintained consistent hyperparameters across all networks and presented identical parameter configurations for the four networks, as shown in Table 5. This approach allowed us to draw comprehensive conclusions and thoroughly evaluate the performance of UD-YOLOv5s in cattle rumination behavior recognition tasks from multiple perspectives. By conducting a rigorous comparative analysis, we aimed to provide robust insights into the strengths and weaknesses of each algorithm, ultimately highlighting the effectiveness of UD-YOLOv5s in this domain.

Table 5

Neural network parameter setting.

Since MEAN-SHIFT is a density-based unsupervised clustering algorithm, MASK-RCNN, YOLOv3, and UD-YOLOv5s are based on CNN architecture, the parameters of the four reconstructed networks are shown in Table 6.

Table 6

Reconfigured network parameters.

5.1.

UD-YOLOv5s Ablation Experiment

We divided the dataset into “rumination” and “other,” represented by labels 0 and 1, respectively. For model training and validation, we utilized a 9:1 partition ratio. The training phase comprised 100 cycles to optimize model performance. To thoroughly evaluate the effectiveness of the jaw skeleton feature extraction method, we conducted ablation experiments under identical experimental conditions. In this paper, using the cattle dataset, we compare and validate the performance of four approaches: MEAN-SHIFT, MASK-RCNN, YOLOv3, and UD-YOLOv5s. The critical evaluation metrics for the models encompass precision (Pre), recall (Rec), frames per second (FPS), mean AP (mAP), model memory usage (MMU), and $F1$ score. Pre denotes the ratio of samples predicted as positive by the genuinely positive model. In the context of cattle rumination recognition, it measures the accuracy of images correctly classified as rumination or non-rumination. Rec signifies the proportion of positive samples the model accurately predicts, indicating the number of samples correctly classified as positive by the classifier divided by the actual number of positive samples. It provides insight into how effectively rumination images are identified. The calculation method for the recall is illustrated in Eqs. (16) and (17). In this comprehensive evaluation, these metrics serve as key performance indicators, shedding light on the models’ abilities to detect cattle rumination behavior accurately. The higher the precision, recall, and $F1$ score, the more reliable and effective the model recognizes cattle rumination patterns. In addition, the FPS, mAP, and MMU metrics contribute valuable insights into each model’s computational efficiency and memory utilization. These evaluations are crucial in determining the most suitable approach for cattle rumination recognition tasks

On the other hand, MMU quantifies the memory size occupied by a trained deep-learning model. In the context of cattle ruminant recognition tasks, MMU serves as a measure of the model’s complexity. A smaller MMU implies a more compact model footprint, making it feasible to deploy the model on smaller devices for inference. This can help avoid overfitting and enhance the model’s generalization performance as it becomes more adaptable to various hardware configurations. The calculation of MMU is typically carried out following the equations provided in Eqs. (20) and (21)

Table 7

The results of the experiment.

5.2.

Evaluation of the Generalization Effect of UD-YOLOv5s

We divided our self-built dataset into two main parts: the training dataset and the test dataset. To assess the generalization effect of the UD-YOLOv5s model, we conducted a thorough evaluation by comparing the accuracy and loss rate between these two sets. The dataset was further categorized into two groups: the regurgitated dataset and the other datasets which were used in the experiment. For the crossover subjects, we ensured that the training and test sets were appropriately balanced, consisting of 16,523 and 7152 samples, respectively. Similarly, for the crossover view, we maintained a balanced distribution of 14,236 samples in the training set and 6482 samples in the test set. Figure 13 illustrates the insightful results derived from our evaluation. The model’s accuracy exhibits a consistent upward trend as the number of training samples increases, reflecting the positive impact of additional training data on enhancing the model’s performance.

Fig. 13

Comparison of the accuracy of the training set and the test set: (a) cross subject (CS) and (b) cross view (CV).

The loss rates associated with the training and test sets in both cross-subject and cross-view iterative training are shown in Fig. 14. The figure demonstrates a notable pattern: as the number of training samples increases, the model’s loss rate consistently decreases. This observation highlights the beneficial effect of utilizing more training data, as it improves the model’s ability to minimize errors and optimize its predictions. This reduction in loss rate reflects the model’s enhanced learning and generalization capabilities, making it more adept at handling different scenarios and achieving better overall performance.

Fig. 14

Comparison of the loss values of the training set and the test set: (a) CS and (b) CV.

5.3.

Performance Comparison

To showcase the outstanding performance of the UD-YOLOv5s model in rumination recognition, we conducted a comprehensive comparative analysis involving three state-of-the-art models: MEAN-SHIFT, MASK-RCNN, and YOLOv3, all of which have demonstrated exceptional capabilities in rumination recognition research. MEAN-SHIFT focuses on tracking the mouth area and extracting mouth features to detect rumination patterns. MASK-RCNN, on the other hand, harnesses the power of the residual neural network to extract mouth area features and trains the algorithm, for instance, segmentation, enabling precise rumination recognition. As for YOLOv3, it processes the image using the backbone network, effectively fuses the features, and produces results through the prediction layer for rumination recognition. Since rumination recognition, which relies on the extraction of mandibular and maxillary skeleton features, is a multi-classification task, we employed the GIoU loss function for model training. To assess the superior convergence efficiency of the UD-YOLOv5s model, we meticulously compared the accuracy and loss values of MEAN-SHIFT, MASK-RCNN, and YOLOv3 at varying training cycle durations, as visually shown in Fig. 15. The results were resoundingly in favor of UD-YOLOv5s, exhibiting a remarkably faster convergence speed while maintaining the same level of recognition accuracy as the other models. When compared at the same convergence speed, UD-YOLOv5s achieved significantly higher recognition accuracy. Consequently, the UD-YOLOv5s model demonstrates unparalleled performance in rumination recognition accuracy when contrasted with the other models subjected to scrutiny.

Fig. 15

Performance comparison of UD-YOLOv5s and other algorithms: (a) accuracy and (b) loss value.

This work is based on the YOLOv5s, and the results of the experimental comparison with YOLOv5s are as follows in Table 8.

Table 8

YOLOv5s comparison experiment results.

Based on the accuracy, recall, mAP, and $F1$ metrics, we conducted statistical tests on a self-constructed ruminant dataset using the following approach:

Data preparation: the regurgitated and non-regurgitated datasets used in the experiment were selected as samples for the statistical test.

Model evaluation: we compared and evaluated the different networks’ accuracy, recall, mAP, and $F1$ scores.

Testing method: considering the nature of the cattle ruminant dataset and the fulfillment of statistical assumptions, we chose the $t$-test as the statistical testing method.

Hypothesis determination: we established the null hypothesis ($H0$) and alternative hypothesis ($H1$). The null hypothesis states that there is no significant difference between the indicators of the two models. In contrast, the alternative view suggests a significant difference exists between the indicators of the two models.

Statistical calculations and significance tests: we performed $t$-tests to calculate and assess the significance of the indicator differences. Based on the chosen method and the calculated results, the obtained $p$-values were used to determine whether the differences were statistically significant.

The experimental results demonstrated that the calculated $p$-value was less than the chosen significance level ($\alpha =0.05$), allowing us to reject the null hypothesis and conclude that the observed differences are statistically significant.

We analyzed the impact of UD-YOLOv5s on the interpretation of actual farm cattle regurgitation videos, as shown in Fig. 16. In the context of recognizing cattle regurgitation behavior in various scenarios, it is evident that the recognition method effectively identifies the cattle mouth area during regurgitation, along with other relevant behaviors, as highlighted by the red box in the figure. UD-YOLOv5s exhibits higher detection accuracy in scenarios without interference from external factors or the presence of cattle pens, overlap, and other complexities within the monitoring area. However, there is room for improvement in accuracy when dealing with such challenging environmental conditions.

Fig. 16

Graph of experimental results of UD-YOLOv5s: (a), (b) other behavior and (c), (d) ruminant behavior.