“Remove this leaf” sounds like a simple instruction. For a robot, however, this involves a complex process of interpretation: It must understand language, identify the leaf in its surroundings and use that information to determine a safe movement. In the new German-Taiwanese research project AgriLoop, a team led by Lamarr researcher Prof. Dr. Maren Bennewitz is investigating how these capabilities, which until now have often operated separately, can work together in an integrated robotic system. The Bonn-based part receives 463,000 euros in funding from the Federal Ministry of Research, Technology, and Space.
From Natural-Language Instructions to Safe Movement
AgriLoop stands for “Learning Expert-Guided Crop Intervention through Human-in-the-Loop LLM Robotics.” The project combines language models with vision-language models and vision-language-action models, along with 3D perception and the physical control of a robot. Vision-language models integrate linguistic and visual information. Vision-language-action models go one step further: they link what the robot perceives and the instruction receives to a specific action. The correct action depends on the condition of the plants and the specific situation in the greenhouse. Experts can, for example, access whether an intervention is necessary and how it can be carried out without damaging the plant. This experience-based knowledge cannot be fully programmed in advance. Put simply, the robot is expected to answer three questions: What task have I been assigned? Which plant or leaf is being referred to? What movement will carry out the task without damaging the plant?
On the German side, the focus is on robotics and 3D perception. Cameras and sensors capture the crop stand, and the resulting data is used to create a 3D model of the greenhouse. The AI links spoken or written instructions to specific objects within this environment. A command such as “Examine this plant more closely” can thus be translated into a targeted and reliable movement toward a particular plant. Within this model, the robot can locate individual plants and plant parts and determine appropriate movements. Real greenhouse environments and horticultural expertise make it possible to develop and test the methods under real-world conditions. The Taiwanese teams are working, among other things, on multimodal language processing, dialogue systems, to linking commands with objects and actions, and learning from human feedback.
The project initially aims to develop a demonstrator that demonstrates the principle of natural-language-based-human-robot collaboration. At the same time, the project is developing efficient AI models that can be deployed directly on the robot platforms without continuous reliance on large data centers.
Why Humans Remain Part of the System
Weather a plant appears healthy, which leaf should be removed and when particular caution is required depend on the specific situation. This experiential knowledge cannot be fully programmed in advance. In AgriLoop, therefore, experts remain involved in the learning and decision-making process: they give instructions, correct the robot, and evaluate its actions.
In this way, the project brings together several lines of research of the Lamarr Institute. As Area Chair for the Embodied AI research area, Maren Bennewitz investigates how intelligent systems perceive their environment, understand information, plan actions, and physically execute them. AgriLoop links this research with human expertise and situational context while also exploring how advanced AI can be operated efficiently on real-world hardware. The greenhouse serves as a challenging research environment for a more fundamental task: robots should not only process expert knowledge but also reliably translate it into actions in changing real-world situations.