Early Prediction of Heat Stress in Dairy Cattle Using Artificial Intelligence for Sustainable Livestock

Overview

Computer vision is a field of artificial intelligence (AI) that focuses on enabling computers to interpret and understand visual information from images or videos. It allows them to extract meaningful insights through tasks such as object detection and video-based decision-making.  A typical computer vision system uses the camera as an input device, which captures visual information in the form of images/videos frames. This input is fed into a machine learning model which is trained using datasets for specific tasks such as identifying and locating objects in an image. The model produces output that provides a visual understanding of the input image or video, which can further be used for various applications.

Recent advances in machine learning applied to visual recognition tasks have enabled the use of computer vision systems in real-world scenarios in various domains, including agriculture and dairy industries. Modern livestock management has evolved significantly in recent years, with a growing emphasis on precision agriculture and the integration of advanced technologies to enhance the well-being of animals and the efficiency of farming operations. In this context, the development of intelligent systems for monitoring and data-driven decision-making has become increasingly vital in the livestock industry. 

This research project introduces a comprehensive approach to precise and personalized cattle monitoring using AI and computer vision for early heat stress prediction. We develop a computer vision system that processes images/videos obtained from a camera installed in a barn, yielding the following outcomes:

1. Detection and identification of individual cows in image/video frame.
2. Mapping of individual cow’s locations onto the ground plane, compatible with both single-camera and multi-camera configurations.
3. Activity recognition of individual cows, such as standing and lying.
4. A comprehensive 2D Barn Map with information on individual cows’ location and activities.

Figure 1 illustrates the proposed methodology. Images/video frames from the cameras installed inside the barn are fed into a computer vision system. It comprises of machine learning (ML) models for cattle detection, individual identification, and activity recognition. The output generated from our computer vision system serves as a crucial source of information, offering valuable indicators closely linked to the occurrence of heat stress in dairy cattle. This personalized and precise data empowers farmers with actionable insights, enabling them to make well-informed and data-driven decisions in a timely manner. This promotes sustainable livestock management and animal welfare.

We now describe our computer vision system in detail. Figure 2 illustrates the design of our proposed methodology. The proposed computer vision system integrates cow detection, identification, and activity recognition machine learning models followed by a geometric transformation to produce a 2D map of the cow locations.

The steps involved in this methodology are discussed in detail below:

Cow Detection

The initial step of our approach involves the detection of cattle in the input images/video using a deep learning-based object detection model as shown in Figure 2 (a). An object detection model outputs (i) a set of bounding boxes that enclose the objects of interest (ii) an object label associated with every detected bounding box (e.g. ‘cow’) denoting what the object is (iii) confidence score associated with each bounding box.

A labeled dataset is required to train an object detector. Each image is annotated with the bounding boxes for the cows present in an image. Bounding box annotations in an object detection dataset are a form of metadata used to define the location and extent of specific objects within images or frames of a video. They consist of rectangles enclosing the objects of interest (e.g. cows) in the images or frames. The rectangle is typically defined using the (x, y) coordinates of the top-left and bottom-right corner of the bounding box. Each bounding box is associated with a specific class label, describing the object it encloses. These class labels are used to identify and categorize the objects present in the image. The  annotations are used as ground truth data during the training of our cattle detection model. The model learns to predict bounding boxes and their corresponding class labels for objects within new, unseen images. When the model predicts bounding boxes on test or validation data, the annotations serve as a reference to measure the performance of the model's predictions.

We employ YOLOv7 (Wang et al., 2022) as our object detection model. It is a state-of-the-art real-time object detector in terms of both speed and accuracy. Images/Videos captured by a camera in the barn serve as the input to our cattle detector model. We train the model for the task of cattle detection using our labeled dataset of dairy barn images and annotations. The dataset was collected using a video surveillance system of two cameras at Emmons Blaine Dairy Cattle Research Center in Arlington, Wisconsin. The dataset consists of a wide variety of cows in terms of location, orientation, occlusion, and postures. Our dataset consists of 2023 annotated images, which were split into train, validation, and test datasets as described in Table 1.

Table 1. The dataset size and splits used for training and evaluation of the cattle detection model

Table 2 lists important hyperparameters used to train the model. 

Table 2. Hyperparameters used to train the cattle detection model

The output of cow detection model provides us with important information on the location of the cows in an image, which serves as input to our subsequent steps in the methodology as shown in Figure 2.

Individual Cow Identification

Following the initial step of cow detection, the next phase in our methodology consists of individual cow identification. Once the presence of cows is predicted in an image through the cow detection process, we isolate and extract individual cow images from the original input image. The bounding boxes obtained from the previous step give us the precise localization of each cow within the image, allowing us to create generate the image crops, each containing a single cow. These individual cow images serve as the input to our identification model as shown in Figure 2 (b).

To accomplish the task of individual cow identification, we employ a convolutional neural network (CNN) as our image classification model. This model is specifically designed to classify each cropped image into one of 16 predefined labels. These labels represent distinct identities assigned to individual cows within the monitored barn environment.

The image classification model, trained on a dataset that encompasses the diversity of cows present in the barn, has the capability to recognize unique features, patterns, or characteristics associated with each cow. This recognition enables the assignment of individual IDs to the cows, facilitating subsequent tracking and monitoring of their activities.

We use EfficientNet (Tan et al., 2019) as our model architecture since it offers a good balance between model size, accuracy, and speed. The classification head of the model is modified such that the last layer consists of 16 nodes to output 16 values between 0 and 1, representing the probability scores for each cow. We train the model using 12,547 annotated images which form the training dataset for our model. These images are obtained by cropping the bounding box regions of each cow in the original images. Each image crop is saved as a separate image file and labeled with the ID of the cow present in the image crop.  Our validation data consists of 1,656 samples which are used to select the best hyperparameters. The model performance is evaluated on 3,086 test images.

Table 3. The dataset size and splits used for  training and evaluation of the cattle identification model

The model was trained for 100 epochs using cross-entropy loss function. The best model is saved by evaluating it on the validation set after each epoch. Important training hypreparameters are listed in table 4.

Table 4. Hyperparameters used to train the cattle identification model

This step of individual cow identification provides not only accurate detection of cattle but also the ability to differentiate individual cows within the herd, enabling personalized monitoring of the cows.

Activity Recognition

Following the steps of cow detection and identification, the next phase in our methodology consists of the critical task of activity recognition. We employ an image classification model trained to distinguish between two fundamental cattle behaviors: "standing" and "lying". As in the previous step of cow identification, individual cow images are cropped from the original input image using the bounding box coordinates generated in the cow detection step as shown in Figure 2 (c). The image classification model is designed to analyze each cropped cow image and determine whether the cow is in a "standing" or "lying" position. This determination relies on the recognition of distinct visual cues and patterns associated with these activities, such as posture, limb positioning, and overall body orientation.

As in the step of individual identification, we use EfficientNet (Tan et al., 2019) as our model architecture. The classification head of the model is modified to adapt to the task of binary classification. It is trained using 2,513 annotated images which form the training dataset for our model. These images are obtained by cropping the bounding box regions of each cow in the original images. Each image crop is saved as a separate image file and labeled with the ID of the cow present in the image crop.  Our validation data consists of 384 samples which are used to select the best hyperparameters. The model performance is evaluated on 736 test images.

Table 5. The dataset size and splits used for training and evaluation of the activity recognition model

The model was trained for 100 epochs using cross-entropy loss function. The best model is saved by evaluating it on the validation set after each epoch. Important training hyperparameters are listed in table 6.

Table 6. Hyperparameters used to train the activity recognition model

The utilization of the activity recognition model not only enhances our understanding of individual cattle behavior but also provides valuable insights into the collective dynamics of the herd. This information is integral for farm management and the early identification of potential issues, including environmental discomfort and health concerns such as heat stress.

The incorporation of activity recognition into our methodology complements the broader objective of comprehensive cattle monitoring. By providing insights into the behavior of each cow, we enable prompt responses to changing conditions, contributing to improved animal welfare and the overall efficiency of dairy farming operations.

Generation of a Comprehensive 2D Barn Map

Building upon the insights and data obtained from the preceding steps in our methodology, we generate a 2D map representing the locations, ID, and activities of the cows within the barn as shown in Figure 2 (c).

The bounding boxes obtained in the first step of cow detection are with respect to the frame of reference of a 2D image. The top-left corner of the image serves as the origin and the coordinates representing the bounding box predictions are measured with respected to this origin in terms of pixels. To translate these bounding box detections into a real-world frame of reference, we employ a technique known as perspective transformation. It is a type of linear geometric transformation used to transform image from one perspective (e.g. isometric view) to another (e.g. top view). As shown in Figure 3 below, perspective transformation maps the ground plane in the barn to a 2D plane in the map. 

This transformation is instrumental in mapping points on a plane from the isometric view of the image frame to a top view that aligns with the barn's actual layout as shown in Figure 3. 

Mathematically, perspective transform is a linear transformation obtained by multiplying the input points with a transformation matrix as shown below.

Since the bottom-left and bottom-right points of the bounding box can be approximated to be lying on the ground plane, we calculate the mean of these points to obtain the location of the cows on the ground plane of the image. Perspective transform is then applied to these points to obtain the location of the cows in the top view. Additionally, we incorporate the unique cow IDs established through the individual cow identification model, as well as the recognized activity labels derived from the activity recognition model. The resulting 2D barn map, enriched with cow IDs and activity information, offers an intuitive overview of the herd's dynamics as shown in Figure 2 (c). This map serves as a valuable tool for farm management, allowing for the efficient monitoring of each cow's location, identification, and activity status. 

Integration of Multi-Camera Setup

While the previous steps successfully yield a 2D barn map for individual camera inputs, the expansion of our monitoring capabilities to encompass a multi-camera setup necessitates a coherent strategy for merging the results and generating a unified 2D map that covers regions of overlapping camera fields of view. This consolidation process ensures a holistic representation of the barn environment, especially when different cameras capture the same cows. Figure 4 shows the images captured by two cameras installed diagonally across the pen, where multiple cows are visible in both the cameras.

In such a scenario where multiple cameras are deployed, some cows appear only in a single camera view whereas the rest appear in both the camera views. To create the consolidated 2D map for such shared areas, we apply a weighted average approach based on the predicted locations of these common cow detections.

The weights assigned to each detection are derived from the confidence scores provided by the detection models. These confidence scores signify the certainty of detections - higher scores correspond to more confident detections. As such, they serve as a basis for determining the contribution of each camera's observation to the final 2D map for the overlapping regions. Thus, the final location of the cow is determined by the weighted average of the locations predicted in the images of each camera, with the weights are the normalized confidence scores for the bounding box detections.

This technique takes into account not only the presence and locations of cows within overlapping regions but also the level of confidence associated with each detection. This results in an accurate and comprehensive representation of cow positions and activities across the entire barn, regardless of the number of cameras used. By integrating data from multiple camera sources and merging it into a unified 2D map, our methodology accommodates a broader field of view, enhances monitoring precision, and provides a more holistic view of the barn environment. This consolidation facilitates the comprehensive management and monitoring of cattle, contributing to improved animal welfare and the efficiency of dairy farming operations.

References

1. Wang, Chien-Yao, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
2. Tan, Mingxing, and Quoc Le. "Efficientnet: Rethinking model scaling for convolutional neural networks." International conference on machine learning. PMLR, 2019.