Collecting image data to improve algorithms


In the last week of March 2017 the Sweeper team has performed tests with advanced colour and 3d cameras on their harvesting robot in the greenhouse of partner Research Station for Vegetable Production at Sint Katelijne-Waver in Belgium. The data of the tests will be used for improving the vision algorithms for detecting ripe fruits and approach path of the robot.

Data was gathered which will be used to improve the maturity detection and increase the robustness of it. Furthermore, a field test has been performed to improve reachability. This means how the robot should approach the fruit to reach it without harming the crop. Improvements will be implemented and in summer new harversting field tests are scheduled to be carried out at De Tuindershoek.

 Robot in greenhouse PSKW


Visual servo control experiments


Earlier this year visual servo control experiments were carrried out on artificial sweet pepper plantsin the laboratories of the University of Umeå in Sweden to investigate the effect of choosing different approaching directions on harvesting times. The value of ideal information regarding the best approach direction with least occlusion from obstacles was evaluated by comparing it to a method attempting several directions until successful harvesting is performed. The results of the experiments will be presented at the TAROS 2017 conference in July.

In this video the robot is trying different approaching directions using visual servo control to locate and move towards the fruits as soon as they are not blocked by leaves.


 sweeper apoproach screenshot


The paper “Design of an eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation”, is listed in the five most downloaded Biosystems Engineering articles from ScienceDirect between January 2016 and July 2016. The publication is written by Ruud Barth, Jochen Hemming and Eldert van Henten and is based on results obtained in the SWEEPER-project. It describes a modular software framework design that allows flexible implementation of eye-in-hand sensing and motion control for agricultural robotics in dense vegetation.

The artile is published Biosystems Engineering, Volume 146, June 2016, Pages 71–84 and can be accessed through this this link.



On February 16th-17th, 2016 the SWEEPER consortium held its 3rd general meeting at the Umeå University in Sweden. Partners presented their progress after the first year of research and development. Umeå University works on the motion control of the robot and gripper, for which it uses simulations and real-time control of the robot. Ben Gurion University worked on the detection of fruit using vision routines. For this work, both universities have a robot arm available. Based upon results of tests of several hard and software modules, a camera, laser and illumination for fruit detection were selected. Next, Irmato assembled a basic version of the sweet pepper robot including the robot arm, a mobile platform and the gripper from the previous project CROPS.



The Sweeper team in front of the building of the Department of Computing Science of the Umeå University.


Currently this robot is placed at the laboratory of Wageningen University and Research Center where it is now under further testing of the several modules. Stepwise, new software from the Umeå and Ben Gurion University will be integrated in order to make the integrated basic version able to pick a sweet pepper. Once the functional tests show that the robot is able to harvest fruits in an artificial crop, it will be transferred to a commercial greenhouse (Tuindershoek) at IJsselmuiden the Netherlands in spring 2016. The Proefstation voor de Groenteteelt in Sint-Katelijne-Waver in Belgium selected the most suitable sweet pepper variety. Now, they started the second season tests to select the best cultivation technique, including LED lighting, for robotic harvesting. In the meantime the Sweeper project has had the first meeting with its end-users represented in the Growers Advisory Board.


First setup of the robot in front of the artificial sweet pepper crop in laboratory of Wageningen University and Research Center.







On September 3rd 2015, the SWEEPER consortium held its 2nd general meeting at the Proefstation voor de Groenteteelt in Sint-Katelijne-Waver in Belgium (PSKW). Partners presented their progress of the first half year. The user requirements, functional and hardware design specifications for picking yellow and red peppers are ready. The PSKW evaluated several yellow and red sweet pepper varieties on their suitability for robotic harvesting and selected the varieties with highest success rates. Fruits are best picked by approaching them from below. Based on that, Irmato selected a manipulator and demonstrated it during the meeting. Ben Gurion University presented evaluations of five camera systems for fruit detection, localization and maturity analysis. Wageningen University and Research Center tested several illumination prototypes and showed that a module with very high light intensity is needed to minimize the negative effect of quickly changing natural light conditions on the image quality. Umea University defined the basic software framework and selected the Robot Operating System (ROS), a set of software libraries and tools, to build the SWEEPER robot application. Prior to the meeting the research team was instructed how to use ROS.


Selected manipulator demonstrated to the consortium during the PSKW meeting


The week after the meeting the manipulator was used to acquire real-world images at the sweet pepper grower involved in the consortium (de Tuindershoek in IJsselmuiden, NL). For this, Ben Gurion University and Wageningen University prepared several camera and illumination system set-ups. Later this year the prototypes with best performance will be chosen to be used to build the basic system. The basic system is planned to harvest the first fruits in spring 2016. The evaluation results will be used to develop the final advanced sweet pepper robot. The upcoming period the team will focus on providing robust sensing functionality to facilitate selective targeting, efficient manipulation and safe harvesting (for the fruit, plant, greenhouse, and robot). The manipulator, having 6 degrees of freedom to move, will be used in the basic robot setup. The available space in between the crop rows is limited and simulations with this manipulator configuration will be performed to check whether the robot can reach all fruits without damaging the crop.


Selected manipulator used to acquire real-world images at De Tuindershoek

At the meeting, coordinator Jan Bontsema announced his departure as off October 1st, 2015, and introduced his successor Jos Balendonck. The whole SWEEPER team thanked Jan for his inspirational leadership during the first phase of the project.


The SWEEPER team at the PSKW sweet-pepper greenhouse compartment








The Kick-off meeting SWEEPER was held on February 12th and 13th, 2015 in Wageningen, The Netherlands

The SWEEPER consortium at Hotel De Wageningsche Berg

The consortium visiting Peppers Unlimited, IJsselmuiden, The Netherlands