Development of an Autonomous Laser-Based Robotic System for Sustainable Weed Management in Vidalia Onions

The development of the proposed robotic laser weeder will be done in three distinct stages. Each stage corresponds to the development of a sub-system or functionality within the overall system. Figure 1 illustrates the complete functionality of the system, highlighting each individual high-level component involved.

Objective 1: Develop a real-time weed detection system for onion fields using RGB and depth data

Onion Field Data Acquisition

The data collection phase involved four visits to the University of Georgia (UGA) Vidalia Onion and Vegetable Research Center in Lyons, GA. The onion planting at UGA Vidalia Research Center occurred in November, and at the time of data collection visits, the plants had reached the 3-5 true leaf stage. This growth stage was specifically selected because it coincided with the weeds reaching their mature vegetative phase.

For data collection, a Zed 2i stereo vision camera by Sterolabs was used which provides high-resolution RGB and depth imagery with up to 2K resolution, recording videos at 15 frames per second (fps). It also offers a depth accuracy of approximately 0.1 mm, which will be useful in finding the location of weeds in 3D space. A mobile data collection system was developed by assembling a structure with T-slotted aluminum frames supported by four caster off-road wheels (Figure 2). The width of the system was kept at 6 feet to match the inter-row spacing of the Vidalia onion fields. The mobile system helped maintain the camera's height and orientation consistently, ensuring maximum stability and ease of mobility.

A graphical user interface (GUI) application was developed using Python and the Tkinter framework to streamline the operation of the ZED 2i stereo vision camera during field data collection. This application enabled real-time control of camera settings and recording functions, including resolution and frame rate. The interface integrated the ZED SDK for hardware control and image acquisition. Figure 3 shows live visualization of RGB and depth streams in the GUI app which helps real-time monitoring of the data quality. Audio feedback signals were also integrated to signal the start and stop of recording sessions for better data capturing in field conditions.

During data collection, continuous video recordings were acquired for each onion row instead of capturing individual static images. This approach enabled uninterrupted spatial and temporal data collection, reducing information loss between frames and ensuring consistent visual coverage across the entire row. With static image data collection, each image required approximately 15 seconds for proper alignment, focus, and recording, including system stabilization. In contrast, video captured at 15 fps results in an improvement of time efficiency by approximately 95%. From the recorded videos, individual frames containing both RGB and corresponding depth information were extracted to train the model.

Data Preprocessing

For video handling and data preservation, FFmpeg (Fast Forward Moving Picture Experts Group) was used to ensure that the depth and RGB data recorded by the Zed 2i camera were saved in their original raw format. FFmpeg is an open-source multimedia framework widely used for recording, converting, and streaming audio and video. It was selected over OpenCV due to its superior performance in handling high-resolution video files and preserving critical metadata. FFmpeg supports direct access to MKV-formatted files recorded by the Zed 2i camera without introducing additional compression or frame drops. Unlike OpenCV, which compresses depth data into 8-bit format during decoding, FFmpeg preserves the depth stream in its native 16-bit "gray16le" format. This format ensures that no quantization error or depth value distortion occurs during processing. Frames were extracted at a fixed sampling rate of 2.3 fps, with a 66% overlap between two consecutive camera fields of view (FOVs).

For data annotation, the CVAT (Computer Vision Annotation Tool) platform was used to label three distinct classes: "weed," "canopy," and "weed_cluster." The "weed" class included all non-crop plant species present within the onion rows, while the "canopy" class referred to the visible above-ground portion of Vidalia onion plants. The "weed_cluster" class was used for dense regions where multiple weeds overlapped and could not be clearly separated. Figure 4 shows an example of the annotated regions. A total of 100 annotations were manually created on selected frames extracted from the RGB video data.

Data Processing

As a starting point for the weed detection system, a comparative analysis of state-of-the-art deep learning models was conducted using publicly available weed datasets. The study evaluated single -shot object detection models including YOLOv9, YOLOv8, RT-DETR, and YOLO-World on two open-source datasets. Dataset A, developed by Rai et al. (2023), contained 3,975 RGB images of four common North Dakota weed species: kochia, common ragweed, horseweed, and redroot pigweed, as shown in figure 5. Images were captured using Canon 90D cameras and DJI Phantom 4 Pro drones in both greenhouse and field settings. The dataset includes over 7,700 annotated weed instances, providing strong spatial and temporal diversity for training. Dataset B, introduced by Sudars et al. (2020), consisted of 1,118 RGB images captured in greenhouse and field environments in Latvia using Intel RealSense D435, Canon EOS 800D, and Sony W800 cameras. It includes 7,853 annotated instances spanning eight weed species and six food crops, with varied growth stages and resolutions to reflect realistic field conditions. The combination of these two datasets enabled a robust evaluation of model generalization and detection accuracy under varying lighting, crop density, and background complexity.