Autonomous robots for agriculture 4.0

|Zuckerrübenfeld in Bonn|Bild des Zuckerrübenfeldes aus Bonn wird übersetzt||||
||© PhenoRob|© PhenoRob|© PhenoRob|© PhenoRob

Crops are an important source of food and other products. In conventional farming, tractors apply large amounts of agrochemicals uniformly across fields for weed control and plant protection. Autonomous farming robots that can sense, understand, and act in the field have the potential to provide environment-friendly weed control and plant protection on a per plant basis. A perception system that reliably distinguishes crops, weeds, soil, and other objects of relevance under varying environmental conditions is the basis for plant-specific interventions such as spot applications. Developing new approaches and techniques, combining new technologies from artificial intelligence (AI), machine learning, robotics, and phenotyping towards sustainable crop production is part of the DFG Cluster of Excellence “PhenoRob.”

To monitor and analyze plants, plots, and whole fields, today’s robots use machine learning techniques combining computer vision and deep neural networks to extract the relevant information from camera images. In research, one often uses so-called semantic segmentation networks that assign a semantic label such as crop or weed to every pixel in each image. Together with information about the robot’s location, for example, from GPS, the environment can be interpreted in a geo-referenced fashion at a cm- to mm-level. Based on such detailed knowledge, robots can take targeted intervention actions, ranging from spot spraying, mechanical treatments, or even laser-based destruction of individual weed plants.

Analyzing and Understanding the Environment

Today’s semantic segmentation systems often show a performance decay when applied under new conditions. This is a general problem in supervised machine learning when applied to real-world problems where the environment changes naturally. Researchers from machine learning, AI, robot learning, and several other fields actively conduct research to bridge this gap between the source domain, for example, the training environment, and the target domain. This transfer of supervised learning systems is often called domain transfer or domain adaptation.

Domain Adaptation within the PhenoRob Robots

Within PhenoRob, we are developing approaches to unsupervised domain adaptation for plant segmentation systems in agriculture. Being able to perform such an adaptation in an unsupervised way allows automatic generalization of the learned neural networks based solely on unlabeled data. Our new techniques aim at generalizing and adapting existing segmentation networks to new field environments, different value crops, and other farm robots. This means, that we can transfer a classifier for sugar beets to other crops, for example, sunflowers or from a field robot to a small unmanned aerial vehicle.

image from bonn looking as from stuttgart - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)
© PhenoRob
The image is the result of our network, in which the image from Bonn is translated to look like an image taken in Stuttgart.

Researchers from PhenoRob have shown that their systems yield a high segmentation performance in the target domain by exploiting labels only from the source domain – across locations, crops, and robots. The techniques make use of CycleGANs. GANs are generative adversarial networks that consist of two neural networks that compete against each other. Through the competition, both networks can get better and better over time, improving their capabilities. What makes the proposed CycleGAN especially effective is the enforcement of / enforcing semantic consistency of the domain transfer by constraining the images to be consistently classified pixel-by-pixel when performing transfers in both directions, for example, from source to target and back to the source domain. This technique substantially improves the transfer of semantic segmentation systems using agricultural fields to new field environments, different crops, and different sensors or robots. This, in turn, impacts the usability of farming robots and has the potential to bring such systems closer to practice, such that the same robots can be used across environments and crops in a robust manner.

The Excelenzcluster already introduced itself in the last post, take a look: Drohnen und Roboter für eine nachhaltige Landwirtschaft der Zukunft

lamarr institute person Stachniss Cyrill e1663922306234 - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)
||© PhenoRob|© PhenoRob|© PhenoRob|© PhenoRob
to the profile

Prof. Dr. Cyrill Stachniss

Cyrill Stachniss is a full professor at the University of Bonn and heads the Photogrammetry and Robotics Lab since 2014. Before his appointment in Bonn, he was a lecturer at the University of Freiburg and with the Swiss Federal Institute of Technology. The research activities of the Photogrammetry and Robotics Lab focus on probabilistic techniques in the context of mobile robotics, navigation problems, and perception. This involves the application and […]

lamarr institute person PhenoRob - Lamarr Institute for Machine Learning (ML) and Artificial Intelligence (AI)
||© PhenoRob|© PhenoRob|© PhenoRob|© PhenoRob
to the profile


PhenoRob is one of six clusters of excellence at the University of Bonn and is carried out together with the Jülich Research Centre. Cooperation partners at national level are the Leibniz Centre for Agricultural Landscape Research (ZALF), the Institute for Sugar Beet Research and Fraunhofer IAIS. The project started on 1 January 2019 and will run until the end of 2025, with funding possible for a further seven years. The […]

More blog posts