Reducing farmer uncertainty in spring forage harvests: image recognition to predict alfalfa-grass stand composition

Reducing farmer uncertainty in spring forage harvests: image recognition to predict alfalfa-grass stand composition

Summary

There is a small range in optimal fiber content (NDF) for lactating dairy cows, making quality-related harvest management decisions critical. The aim of this project is to improve the timing and nutritive value of spring forage harvests for dairy operations by reducing uncertainty in stand composition. Mis-estimating composition by 20% can result in late harvests by five or more days, potentially leading to NDF at harvest > 5 g/kg past target levels. This represents critical potential nutritive and economic losses for dairy farms. Accurate prediction equations exist for estimating NDF content of mixed alfalfa-grass stands in spring, and estimating the optimal harvest date. The weak link is estimating the proportion grass in a stand. In spring 2011 we acquired 580 digital images of alfalfa-grass stands in farmer’s fields and determined alfalfa and grass dry matter percentages for vegetation in each image. A second set of 180 samples was gathered in spring 2012 with four digital cameras. A program was developed to estimate alfalfa-grass proportions. The program used digital image processing to filter 64 x 64 pixel tiles from original images. These tiles were transformed to the frequency spectrum using the Fast Fourier algorithm, and selected frequencies were aggregated and supplemented by alfalfa maximum height and grass canopy height data for artificial intelligence processing. Tiles in a subset of selected images were classified as predominately grass, alfalfa, or non-classifiable. Predicted stand composition by supervised Support Vector Machine trained with 1,000 to 15,000 tiles significantly predicted actual values (p < 0.0001, n=548) when all grass species were pooled. Best results were obtained with a set of 4000 training tiles (r2=0.39). Species-specific predictions were better than overall results for timothy (r2=0.5) and reed canarygrass (r2=0.44). A timothy-specific training and testing set was also undertaken with better results (r2 > 0.50). The next step is to continue grass species-specific training and predictions with refined image filters and Support Vector Machines. When satisfactory predictive capacity is achieved, the only required inputs to an Internet program accessible by computer or smart phone include an image of the stand and agronomic measures. Farmers and consultants could use this technology to prioritize the order of harvest of alfalfa-grass fields to maximize chances of obtaining optimal forage NDF for lactating dairy cow diets.

Objectives/Performance Targets

Ultimate Objective:
-Improve the timing and quality of spring forage harvests for Northeast dairy farms by reducing uncertainty in the estimation of alfalfa-grass stand composition.

Proximate Objectives:
1) Capture digital images from representative samples of mixed stands of alfalfa-grass in Northeast farmers’ fields.

-180 representative samples of mixed stands of alfalfa-grass in Northeast farmers’ fields were gathered in May 2012 using four different digital cameras.

2) Determine known stand composition values for each sample.

-The 180 samples of alfalfa-grass were cut, separated into alfalfa and grass fractions, and dried to determine known stand composition percentages

3) Create an alfalfa-grass recognition system that can filter and normalize a mixed stand image, evaluate the image, and return the percentage grass and alfalfa in the stand.

-Image processing techniques were developed and tested on the 580 samples obtained in spring 2011. An alfalfa-grass recognition program consisting of these complex image filters, feature extraction algorithms, and artificial intelligence (AI) has been designed and tested on the 2011 dataset. It will be further refined to improve results, and retested on both the 2011 and 2012 datasets.

4) Generate a free web service on http://www.forages.org that will allow farmers to upload images acquired from their fields and receive stand composition results and predictions of optimal forage quality and harvest timing in a rapid manner.

-This objective will be undertaken following successful completion of the alfalfa-grass recognition system.

Accomplishments/Milestones

Certified Crop Advisors in Tompkins and Cortland Counties of New York State assisted in the selection of appropriate farmers’ fields for sample acquisition. In each field, representative samples were selected and delineated using a 66 cm diameter round hoop. Digital images were acquired at five-megapixel resolution using four digital cameras (Canon Powershot A3100 IS, Olympus FE230/X790, Nikon Coolpix S6200, and iPhone 4 (including high dynamic range (HDR) duplicate images)). In each sample, alfalfa maximum height, grass maximum height, and grass canopy height were measured using a meter stick.

Following digital photo acquisition, a 10 cm high quadrat was inserted under the hoop to define the cutting height and the hoop was lowered to ground level. The forage within each hoop (sample) was cut at 10 cm above ground level by cutting at the height of the quadrat using battery-operated grass clippers. Harvested forage from each sample was labeled, transported back to the laboratory, and manually separated into alfalfa and grass fractions. Any plant species other than alfalfa and grass were removed from the sample. Samples were dried until stable weight was reached at 60 degrees C in a forced air oven. Samples were weighed directly after removal from the oven to determine dry matter weight of alfalfa and grass fractions from each sample. Dry matter weights determined known percentages of alfalfa and grass associated with each sample and its corresponding digital image.

A total of 180 samples were acquired (840 images including: 180 Canon, 180 Nikon, 180 Olympus, 180 iPhone 4, and 120 iPhone 4 HDR). Samples included three varieties of grass (orchardgrass, reed canarygrass, and timothy) that were selected based on frequent establishment in mixed stands with alfalfa in Northeast fields. This sample set complements the 2011 set of 580 samples and will be used to test and validate the alfalfa-grass recognition program. The large set of images taken with multiple cameras, multiple grass species, and in variable field conditions, lighting, stand compositions, and years will help ensure the development of an alfalfa-grass recognition system that is robust under different conditions.

In early 2012, algorithms were designed to automate the hoop extraction and image filtering process. The hoop extraction process has a current success rate of approximately 80%. These processes are now in a testing phase and are being refined to improve performance. Initial image filtering consists of extracting all of the green pixels, removing all low value pixels, finding the left, right, top, and bottom pixels of the white hoop, and then extracting the hoop (Figure 1). The extracted circular image is then converted to a gray scale with an emphasis on green pixels (Figure 1).

Following hoop extraction, multiple image processing techniques were developed and combined with artificial intelligence approaches to generate stand composition estimates (Table 1). The application showing the most potential for success is a tile extraction method with fast Fourier transformation (Polder et al., 2007) (Figure 2) combined with Support Vector Machine (SVM) processing using the LIBSVM open source package (Chang and Lin, 2011). Simpler techniques such as color separation and blob detection were not feasible for this application due to the complexity of mixed stand images. For example, the shades of green in alfalfa and grass are too close to estimate stand composition based on color differentiation. Blob detection was infeasible given broken lines at the pixel resolution level, thus blobs (leafs) could not be defined as distinct entities for quantification during image processing.

Our current approach, tile extraction with fast Fourier transformation (Figure 2) and SVM processing consists of the following steps: 1) Tile extraction (Polder et al., 2007) was applied to the extracted circular gray scale image to crop 64 x 64 pixel tiles from the image (Figure 2). Approximately 400 – 600 tiles are extracted from each original image. 2) The fast Fourier transformation was applied to individual tiles for conversion to the frequency spectrum (Figure 2). 3) All frequencies under a threshold value of 175 were ignored for data aggregation and fast processing speed. 4) Six axial frequencies were aggregated along both the x and y axis for processing by SVM. 5) Tiles within a subset of all 2011 images were binary-classified as predominately grass (1), alfalfa (0) or unclassifiable. Initially, 6% of the original tiles from 2011 images were classified to test the supervised SVM approach (SVM attempt 1 and 2). Subsequently, all timothy images where hoop extraction was successful were classified and used in species-specific supervised SVM development (SVM attempt 3). Classified tiles were used in SVM training using the LIBSVM open source package (Chang and Lin, 2011). 6) Between 1,000 to 15,000 tiles were used to train each SVM run. 7) Each trained SVM was applied to predict stand composition for 2011 samples and compared to known values to assess system performance (Table 2).

Multiple SVM iterations have been undertaken including a SVM trained with tiles from the full 2011 data set and tested on all grass species (Table 2, SVM attempt 1) and on individual grass species (Table 2, SVM attempt 2). A grass species-specific set was also attempted with timothy (SVM attempt 3). The SVM was trained with 3,000 to 10,000 tiles from 47 randomly selected timothy images and tested on remaining 48 samples not used in training (Figure 3). Only samples with good hoop extraction were used. Species-specific results for timothy are better than those achieved previously by visual researcher estimation alone (Parsons estimated grass fraction and determined known values for nearly 600 samples in 2004 (y = 0.22 + 0.69x, r2=0.43, RMSE= 0.147)). The SVM method is now being revised to improve predictive capacity.

The filtering, tile extraction, and SVM system is now being refined for full testing and validation using the 2011 and 2012 datasets. After finalizing refined filters, binary tile classification must be completed for the 2011 and 2012 datasets. This time intensive step requires approximately 10 – 15 minutes per original image. The SVM system will be fine-tuned using the classified set to generate a robust final product. A small 2013 validation dataset will also be collected to further evaluate the system and ensure robustness across years and field conditions. Alternatives to the fast Fourier transformation that could be applied individually or in combination will also be considered (e.g., linear binary patterns, wavelet transformation).

Once satisfactory capacity to predict stand composition is achieved, the alfalfa-grass recognition system will be finalized as a farmer-friendly web application for inclusion as a free web service on http://www.forages.org. The system will consist of six modules: 1) Web interface, 2) Recognition controller, 3) Image filters, 4) Feature extraction, 5) Classification system, and 6) Results. All modules except the web interface have been developed and are now being improved in an iterative testing, validation, and reprogramming process. The image recognition software will be linked to automated calculators that employ equations to determine optimal forage quality and harvest timing (Parsons et al., 2006). Farmers and extensionists will be able to acquire digital images from representative samples of mixed stands using a camera or smart phone, upload images acquired from their fields to the web application as well as the alfalfa maximum height parameter, and receive stand composition estimates and harvest timing predictions to achieve desired forage quality from mixed stands at a given stubble height.

Chang, C.-C. and C.-J. Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27.

McRoberts, K.C., Cherney, J.H., Benson, B.M., and Cherney, D.J.C. 2012. Image recognition to predict alfalfa-grass stand composition. Oral presentation of abstract at 2012 Joint North American Alfalfa Improvement, Trifolium and Grass Breeders Conference, Ithaca, NY.

Parsons, D., Cherney, J. H., and Gauch, H. G., Jr. 2006. Estimation of Preharvest Fiber Content of Mixed Alfalfa–Grass Stands in New York. Agronomy Journal 98:1081-1089.

Polder, G., F.K. van Evert, A. Lamaker, A. de Jong, G. van der Heijden, L.A.P. Lotz, T. van der Zalm, and C. Kempenaar. 2007. Weed detection using textural image analysis. In 6th Biennial Conference of the European Federation of IT in Agriculture (EFITA). Glasgow, UK.

• Figure 1
• Table 1
• Figure 2
• Table 2
• Figure 3

Impacts and Contributions/Outcomes

2012 objectives were achieved by obtaining a large second dataset with multiple species of grass using multiple cameras. Initial iterations of the alfalfa-grass recognition system have been tested and are currently being refined. When complete, the web system will be freely available for farmer and extensionist use to accurately estimate mixed stand composition and help reduce uncertainty in spring forage harvests. Despite challenges in developing high performance image processing and artificial intelligence methods in 2012, we are optimistic that our current tile extraction and Support Vector Machine approach will result in an application that will be useful for reducing farmer uncertainty in the timing of spring forage harvests to ensure high quality, thus improving agricultural sustainability for Northeast dairy farms.

Multiple outreach activities and conference presentations were completed in 2012:
1) A progress report entitled “Reducing farmer uncertainty in spring forage harvests: Image recognition to predict alfalfa-grass stand composition” was presented at the Field Crop Advisors Cornell Cooperative Extension retreat in March 2012.

2) A progress report was presented at the annual Northeast Certified Crop Advisor training sessions to help increase awareness about the usefulness of this upcoming technology.

3) Method development and results of the Naive Bayes Classifier Artificial Intelligence approach were presented at the 2012 Joint North American Alfalfa Improvement, Trifolium and Grass Breeders Conference.
-McRoberts, K.C., Cherney, J.H., Benson, B.M., and Cherney, D.J.C. 2012. Image recognition to predict alfalfa-grass stand composition. Oral presentation of abstract at 2012 Joint North American Alfalfa Improvement, Trifolium and Grass Breeders Conference, Ithaca, NY.

4) Method development and results of Support Vector Machine approach were presented at the American Society of Agronomy Joint International Annual Meetings.
-McRoberts, K.C., Cherney, J.H., Benson, B.M., and Cherney, D.J.C. 2012. Reducing farmer uncertainty in spring forage harvests: digital image analysis and artificial intelligence to predict alfalfa-grass stand composition. Abstract accepted for poster presentation at American Society of Agronomy Joint International Annual Meetings, Cincinnati, OH.

Collaborators: