Computer vision algorithms to help decision-making in cattle production

• Computer vision represents a valuable tool for helping cattle producers make decisions. • Deep-learning algorithms, especially neural networks such as convolutional neural networks, conduct image classification, segmentation, object detection, and feature extraction. • Computer vision helps to estimate intake, body weight and gain, body condition score, health status, and reproductive performance of cattle. • The main goal for the future is to set up computer vision on-farm to execute real-time algorithms. In recent years, integrating computer vision technologies into precision livestock farming (PLF) management systems have the potential to transform how cattle producers collect, monitor, analyze, and optimize animal production. Livestock production, particularly in cattle farming, encompassing beef and dairy under intensive and extensive production systems, faces numerous challenges ranging from optimizing feeding practices to detecting and managing diseases. Traditional monitoring and assessment methods often rely on manual labor and subjective evaluations, leading to inefficiencies and inaccuracies. Computer vision (CV) algorithms, leveraging the power of artificial intelligence (AI), machine learning (ML), and deep learning (DL) offer unprecedented capabilities in automating tasks, extracting meaningful insights, and improving animal management in different production systems and assisting in the decision-making processes such as 1) improving economic impact through solving inaccurate feed inefficiencies (Tedeschi, 2019), 2) decreasing productivity losses due to early disease´s detection in dairy cows (Miles, 2009), and 3) improving labor efficiency by automating tasks and providing real-time insights. Although CV is applicable to all livestock animals, most of the current development of algorithms for usage are conducted for swine production, given the availability of resources and the relatively straightforward CV application for recording indoor-housed animals. In this review, we aim to illustrate the use of cameras and the most utilized CV algorithms applied to cattle production, with a focus on their applicability in predicting key parameters such as individual feed consumption, feeding behavior, body weight (BW), body condition score (BSC), and the detection of prevalent cattle diseases including laminitis, mastitis, and heat stress (Figure 1). Furthermore, we discuss the advancements in cattle identification using CV techniques and their implications for enhancing the traceability and management of individual animals within the production system. By synthesizing the existing literature and highlighting the most applicable CV algorithms, we seek to provide insights into how these technologies can be applied on-farm to help decision-making processes for cattle producers and identify potential avenues for future research and technological innovation in PLF. Schema that summarizes the placement of main cameras and the applications of the main computer vision algorithms in cattle prediction. Computer vision algorithms are based on deep-learning neural networks, which are inspired by the functioning of the human brain through the connection of neurons to execute complex activities that require rationing (LeCun et al., 1988). The deep-learning architecture is composed of a sequence of multiple layers composed of neurons that transform an input into an outcome. A general neural network is usually shown with lines representing the weights and circles indicating neurons, where each neuron is one value, just like every input and output neuron is one value. In the feed-forward pass, each neuron is calculated with equation (1) as follows: where Ok represents the output of the neuron k; I is the previously connected neuron; w is the weight parameter, bk is the bias term for the neuron k, and σ is an activation function to ensure a positive output. In simpler terms, the formula describes how a neuron processes its inputs to produce an output. It takes the weighted sum of its inputs, adds a bias, and then applies an activation function to determine the final output. This process enables the neural network to learn and make predictions based on the input data. One layer (vertical circles) can have many neurons and is fully connected with the previous input. The key to a successful neural network is the proper weights and biases in the model. The optimization of the collection of all weights w→ is called back-propagation, and the same applies to biases. A deep neuron network (hence the name DL) might have millions of weights and biases with hundreds of layers. These DL algorithms’ main goal is to minimize the estimated outcome’s prediction error through iterative back-propagation steps, which modify the weights of connections between neurons (Lee et al., 2018). In each iteration, the difference between the prediction and the ground truth label (i.e., the observed values) measured using a loss function of f(prediction,label) called gradient descendent (∇). This gradient ∇ is a collection of partial derivatives concerning each weight (LeCun et al., 1988), written as [∂f(w→ )/∂w11,∂f(w→ )/∂w21,…]⁠. This function has the same dimension as w→ such that the new w→ can be updated using equation (2), as follows: where LR is the learning rate, the ∇ is the gradient descendent function and w→ is the vector containing all parameters and/or variables (weights). In simpler terms, the formula describes how to update a model’s weights during training. First, calculate the gradient by computing the gradient of the loss function concerning the current weights; this gradient indicates how the loss function changes with small weight changes. Second, adjust the weights: subtract the product of the gradient and the learning rate from the current weights; this step moves the weights in the direction that reduces the loss function. This iterative process aims to find the optimal weights that minimize the loss function, thereby improving the model’s performance on the given task. In equation (2), the learning rate is a tiny number, such as 0.0001. After many iterations for training, these neural networks should be able to produce outputs close to the correct labels. Neural networks are highly flexible and can approximate any nonlinear relationships between known inputs and outputs. Given the nature of the CV input format, which is a matrix with 𝑤×ℎ×𝑐 dimensions, and the output format, which is typically one dimensional as probabilities and regression, the convolutional neural network (CNN) is the most successful so far because convolutional filters extract and preserve unique information. A general CNN model consists of different layers that compute convolved output, and it is also able to change image scale or size with max pooling. The fully connected layers produce the output the same way as the regular neural network mentioned above. Typically, thousands of different convolutional kernels (filters) are applied and optimized. The main functions of these DL algorithms in CV include (Figure 2). Example of different deep-learning algorithms used in computer vision over the same image. Each output neuron in a neural network represents either the probability that an image belongs to a specific class or a numerical value. The collection of these probabilities corresponds to the predefined classes (e.g., class1, class2). Typically, the class with the highest probability is selected. The loss function commonly used is cross-entropy, which measures the difference between the actual distribution of the labels and the predicted distribution output by the model. In the case of regression, the output neurons are the targets, and the loss function is a differentiable equation measuring the numerical differences, e.g., mean square error (MSE). This model structured for regression can be the same as classification, with convolutional (CONV) and max pooling layers, with the last output being numerical values such as age or BW. Successful classification/regression CNN models include ResNet, VGG series, and vision transformer. They established large classification datasets such as ImageNet or CIFAR. It represents the categorization of the algorithm into region proposal- and regression-based methods. In the former approach, object regions are proposed for one or more categories in an image, whereas the regression-based method detects objects by framing their coordinates as a regression problem. One of the earliest prosperous region proposal-based techniques was the region-based CNN (RCNN). This algorithm generates a multitude of region proposals using selective search, followed by classification for each proposed box. The Algorithm you-only-look-once (YOLO) used anchor boxes to generate box proposals followed by a probability-based filtering process. Also, large object detection datasets include COCO and Objects365. It consists of a CV technique that involves dividing an input image into meaningful regions or segments based on the semantic content of the image, and the primary goal of this algorithm is to classify each pixel in the image into a specific category or class. This image segmentation constitutes a foundational component of numerous CV frameworks, entailing the division of images into distinct segments or objects and assigning recognized classes to these segments. Some successful segmentation models include SegNet, U-Net, and PSPNet. It consists of identifying and representing distinctive structures within an image. The intermediate features from the neural network middle layers are beneficial in categorizing objects and linking image content to pre-existing consolidated knowledge. For instance, classification and regression networks can discern recognized biological or structural features within images. Encoder–decoder networks excel in identifying the most compelling feature set describing a collection of images. Computer vision algorithms could be categorized based on their different input and output formats. The input for the CV system can be both 2-dimensional digital images (matrices) with multiple channels and videos. The most common types of images are red, green, and blue (RGB) colored images and extra channels, which include inferred or depth, which can provide shape and temperate information. However, images can only provide signals in a single instant. Videos, made of consecutive images (frames), can support more tasks because of the richer time-based information. On the output side, a typical task for CV is classification, which is a list of probabilities, each corresponding to a specific class. These probabilities can be mutually exclusive (one class per image) or multiclass classification (multiple classes for one image) for both images and videos, for example, lameness, feeding, and sickness. Other outputs in CV are the regression numbers such as BW and age, and others can be designed with one or several output neurons with the same intermediate neural network structures. Both classification and regression are well-studied in the CV community, and algorithms, including ResNet and vision transformer, can perform well in many tasks. Moreover, combining both can detect objects with bounding boxes or pixel segmentations. In addition, beyond probability and regression, 3-dimensional point cloud reconstruction of animals can support detailed measurement and animal re-identification. Object tracking techniques are often integrated with object detection to identify individual animals. Altogether, the categorization in Table 1 groups CV structures with similar input and output formats so they can potentially be re-used for transfer learning. Transfer learning, which is widely adopted in CV applications, is an effective way to deal with small datasets that are common in livestock The is to a model on large datasets with millions of such as COCO or most of the weights and structures be for specific tasks because it is and to transfer learning to a new model as the model the features of image and outputs of computer vision algorithms used in cattle with their functions convolutional neural region-based convolutional neural only and outputs of computer vision algorithms used in cattle with their functions convolutional neural region-based convolutional neural only In cattle farming, the of CV algorithms is These algorithms offer to and animal within the precision livestock et al., By the power of image techniques and machine learning algorithms, CV systems can tasks, ranging from individual animal monitoring to systems for measuring feed have such as in feeding to the feed by individual have utilized these et al., However, both systems and are to research dairy This is to their and the for for most CV algorithms with an of helping to on-farm individual in cattle. For different methods the of the feed or in the and the of feed from the feeding image algorithm can be to the weight of the of feed cameras have conducted through including structured detection and ranging methods et al., and the of cameras under to from in particularly constitutes a within the machine vision tasks such as classification, detection, and et al., (Figure 3) have conducted in the of predicting individual from the feed by using CNN to et mean error and and in the feed of dairy et utilized a transfer learning model based on an of and per with a of mean error of and per for the different in that in a of to et used a network to measurement of feed for dairy cows by with an and an of and in the of to Example of using computer vision in cattle to individual feed A et al., and et al., individual through the of feed in the et al., and et al., through the of feeding or feeding On the to estimate individual through the of feeding by predicting feeding through et on the feeding prediction representing an of and and an and for the of mean mean between and feeding using the et used the to determine individual feeding activities and an precision for all activities of and in and After feeding the prediction of the individual in highly However, using this it should be the of all prediction of the feeding activities error of the from feeding and error of from predicted which the of the In addition, that impact the of these algorithms have to be such as or et al., Computer vision algorithms have also proposed as a method for both individual BW and cattle weight In this most of the conducted on extracting animal from cameras or images et al., for the BW from these et al., The most utilized regression algorithms to BW from body are regression, support vector regression to the classification the gradient regression, the and the most recent the gradient and the gradient Other have utilized one or of these algorithms with cameras in both and (Figure et estimated body and and then predicted BW from digital an of and et BW from the of animals and the of an of and they established a multiple BW et estimated BW from digital images through a an of In to make this task more et multiple CV algorithms networks, and models with to BW BW being the CNN the one the highest performance in BW prediction et utilized similar CV algorithms and cameras cameras to the previous from a BW predictions Also, et the of the prediction of BW through images using artificial neural networks in beef the the different of an and for the BW an and for and and for the of BW recent to BW from the images including body et several regression algorithms to BW from a Also, et utilized these regression algorithms to estimate BW with predicted from an of and an of used DL models to BW. et prediction with a mean as as in dairy et predicted BW with mean error of through the DL algorithm in beef cattle. Example of using computer vision in cattle to body weight or are shown in A et al., et al., et al., and et al., the CV algorithms have their of predicting BW of are that be to be able to the potential of these including 1) relatively small numbers of animals per 2) different and 3) use of measures and (e.g., and different different and and the by is typically by a through or this manual and is Moreover, the are to and on the is a for and In the current livestock based on and are (Figure to on cattle body parameters for as a is commonly typically features such as or body are a regression model is using these either through manual or computer et al., In addition, prediction is highly on the as and et on the images and utilized features from these to an of et shape and regression to a mean error of et the method to detect the and that the method a classification of and a of images. Example of using computer vision to estimate body condition score in cattle. are shown in A et al., et al., 2019), and et al., However, more have as a for richer body to or systems et al., one of the for is the of The cameras or with pixel the and measuring its In cows using a and features from the and regions in predicting within a from the actual to have a between the ground truth score and the estimated et utilized a to images of cows as they it then applied the model to estimate an of within a from actual et parameters for prediction using regression a of and a of et used the and features from images as with class of the of the advancements in systems and mentioned it is to that vision only a 2-dimensional animal The of the dimension in vision applications that rely on et al., advancements in systems and is a complex term encompassing all health parameters such as diseases laminitis, and or heat stress (Figure we with one of the most in cattle production, the (Miles, during a health For et the which to the of and utilized it to detect in the for Other utilized algorithms that segmentation and of an of et al., et used a of the algorithm to identify cows and determine an of and an of in cattle production is laminitis, which has through the of et al., 2018). This is by changes in where the method was adopted to the body and determine the the difference method was used to identify the et al., 2018). on this the proposed an algorithm that this and using the shape by the of the images as a feature for extracting the of the Example of using computer vision to health in cattle. shown in A et al., et al., et al., and et al., mastitis, the leading health in dairy et used a method that the and difference detection method with the through an and of detection of and heat a where they utilized the to the of several body such as the or and this with rate, and This predicted heat stress through rate, which a and of with Other such as reproductive have also through CV algorithms in cattle production. these reproductive detection to determine the to cows has as a highly the economic of cattle. et utilized a object detection algorithm with CNN models to reproductive through bounding identifying of cows in CV algorithms have their capabilities in predicting variables to help make cattle production, impact the of these One is the of and models on new In CV applications, it is to ensure that models can to datasets or different animal and However, due to the in a model on new datasets a This is particularly in precision livestock farming based on CV and 2018). Also, we can find the that we to estimate is such as or be using as many are In such are models models with functions for the Other are to the algorithms by such as representing any techniques of that are of known that are for with the prediction et al., these to cattle production should also be such as the changes in up precision livestock farming, it is to like and as they can impact cattle farming and For instance, the to or can in or This to the mentioned and such as The of in the cattle particularly in the of such as computer vision In the of cattle farming, are just an a as the effective management of such on human Moreover, the of and that is well to specific the in precision livestock the faces in and research into that can be applied in the However, a more the and of within the of animal This is particularly given the made in both large and small production over the last years, by in and (Tedeschi, This between advancements and application to be a to the of CV and technologies in animal main future direction in livestock production is to help the efficiency and of animal production. these terms, we CV as a valuable tool to real-time and on-farm data. CV algorithms should be of on-farm in which be used to algorithms helping of cattle such as the most weight of animals, precision techniques to an of animal on animal performance through early detection of health and diseases and a more to the of the For research should be only on improving the of these algorithms also on of predicted performance to determine to which CV algorithms could be applicable to it is to ensure that the of and is only in and CV also in the biological of animal production (Tedeschi, This focus to more CV applications that are integrated with biological insights. research only on the of these algorithms and to the applicability of CV algorithms under also on These should aim to with a of both the technological and biological of animal a to future in livestock A of computer vision (CV) algorithms are to cattle performance such as intake, gain, health status, and reproductive neural networks, particularly convolutional neural networks (CNN) and neural networks have highly effective in predicting these performance However, the of this on faces including a of for and enhancing the algorithms and like these is these algorithms only to animal performance also to so in real-time and under actual on-farm This the use of such to cattle with enhancing both the efficiency and of their development and of these technologies are to fully their potential in improving livestock is a research in the of in animal and In on computer vision in the of to animal performance through research on how to individual collection of cattle to determine performance and animal which be used to precision algorithms. main is to these algorithms on-farm to help decision-making of cattle by improving their feed efficiency and decreasing their is an from the of in the of is also the of the and the for the of in from in in is a of the for the is an and a to the is the of the research include digital machine learning, numerical algorithms, and is an research the of a previously as an in from to and as a in the of to a in from an of from of and a in from is a in research on computer vision applications in precision livestock farming, and is a research in the of in a in animal from in and a from the of in such as and and as a and to numerous in the and with precision livestock research on optimizing beef cattle production through leveraging and to productivity and is a and the of is also an the of a in and a in and from the of and a in from research focus on the of of and the development of support for the cattle involves integrating and using in and in combining cattle and management with artificial is recognized for in of production. has made to the through to in cattle. The in this are of the and the or of the or the of the main images and of the and the main and the main and the main and and the content and final The of and the from the and the and in this