Multiple camera fruit localization using a particle filter

Apart from socioeconomic factors, success of robotics in agriculture lies in developing economically attractive solutions with efficiency comparable to that of the humans. Fruit localization is one of the building blocks in many robotic agricultural operations (e.g., yield mapping and robotic harvesting) that determines 3D Euclidean positions of the fruits using one or several sensors. It is crucial to guarantee the performance of the localization methods in the presence of fruit detection errors and unknown fruit motion (e.g., due to wind gust), so that the desired efficiency of the subsequent systems can be achieved. For instance, inaccurate localization may severely affect fruit picking efficiency in robotic harvesting. The presented estimation-based localization approach provides estimates of the fruit positions in the presence of fruit detection errors and unknown fruit motion, and it is based on a new sensing procedure that uses multiple (⩾2) inexpensive monocular cameras. A nonlinear estimator called particle filter is developed to estimate the unknown position of the fruits using image measurements obtained from multiple cameras. The particle filter is partitioned into clusters to independently localize individual fruits, while the behavior of the clusters is manipulated at global level to maintain a single filter structure. Since the accuracy of localization is affected by errors in fruit detection, the presented sensor model includes non-Gaussian fruit detection errors along with image noise. Fruit motion can significantly reduce harvesting efficiency due to errors in locating moving fruits. In contrast to existing methods, the dynamics of fruit motion are derived and included in the localization framework to obtain time-varying position estimates of the moving fruits. A detailed theoretical foundation is provided for the new estimation-based fruit localization approach, and it is validated through extensive Monte Carlo simulations. The performance of the estimator is evaluated by varying the design parameters, measurement noise, number of fruits, amount of overlap in clustered fruit scenarios, and fruit velocity. Correlation of these parameters with the performance of the estimator is derived, and guidelines are presented for selecting the design parameters and predicting performance bounds under given operating conditions.