Children have the wonderful habit of asking the most uncomfortable question: why? Why is the sky blue? Why do birds fly in a V? Why does the cat move its tail? And, as it turns out, “why?” is also the question I hear most often when I explain my own research. I work on deep learning for animal behavior. That usually sounds exciting at first: AI and animals. And I go on explaining: I use methods to track movements of animals through time, ensure our recordings are reliable, and extract information such as the direction of movement or action the animal is doing.
The follow-up question is almost always: “Okay… but why does it matter?” Why should we care if a model can track a mouse more accurately? Why do we need AI to tell us whether a fruit fly turned left or right?
To answer that, we need to take a step back.
Why are we interested in animal behavior at all?
The first answer for studying animal behavior is to understand how animals survive, reproduce, communicate, cooperate or compete, care for offspring, or adapt to changing environments. Behavior is our most direct window into these questions. Watching what animals do, tells us how they face these problems. This is the classical territory of ethology: the study of animal behavior in naturalistic settings. This matters for scientific curiosity of course (who has never been mesmerized in front of an animal documentary?), but also for biodiversity conservation. A species changing its feeding patterns, mating rituals, or social structure, may be an early warning sign that its environment is changing. Occasionally, this can also inspire us to find better solutions to some problems: ants can help us find better paths in complex route maps (Ant Colony Optimization algorithms), or quadrupeds can help us build more stable and energy-efficient robots (work from Owaki & Ishiguro).
The second answer lies in the relationship between the brain and behavior: ultimately, behavior is the final output of the brain. If ethology asks how animals behave in the world, neuroscience asks how the brain makes that possible. How does a mouse decide whether to explore or stand still? How does a fly change its flight direction? How does a fish find a mate? How is the world and the stimuli inside it perceived? Which neurons process and transmit the information? How is this information transmitted back to the body to move and act upon the situation?
These are not only fascinating questions in basic science, exploring these also helps us understand the human brain and its neurological disorders.
Because neuroscience aims to understand the nervous system, behavior is often used as a way to observe the effects of changes in the brain. Now that we understand the reasons behind why we want to measure behavior, …
How do we measure animal behavior?
Historically, the answer has depended on the field.
Ethologists often aim for a comprehensive vocabulary of behavior, and list all observed actions in an ethogram, a structured description of behavior. An animal may be resting, locomoting, grooming, courting, feeding, threatening, retreating — and each category can branch into more specific sub-actions. An ethogram is not just a list, it lets researchers describe which behaviors occur, how often they occur, how long they last, and in what order they appear. This approach treats behavior as something worthy of description in its own right.
Neuroscience, in contrast, has often relied on simplified behavioral measurements. These measurements were mainly done using anesthetized animals or controlled stereotypical behavioral paradigms. With anesthetized animals, researchers could gain a clearer picture of the brain, but observable behaviors were limited to passive movements, such as whisker twitches or pupil dilation. Stereotypical behavioral paradigms were designed to test specific capacities, such as sensory processing, learning, or anxiety in a standardized and reproducible manner across experiments and laboratories. In both cases, the aspects of behavior that were measured were predetermined by the experimental design, for example: time to reach a reward zone, pupil dilation in response to a stimulus, or the number of rewards obtained across trials.
These measures are incredibly useful. The classic lever press, for example, became a powerful way to study learning, reward, motivation, and decision-making. Simply counting how often an animal presses a lever can show how an animal learns that an action leads to food, avoids punishment, and changes under neural manipulations. Neuroscience has become extraordinarily powerful at measuring and manipulating the brain, but too often in isolation from realistic behavior. The gap between the incredibly precise description of neuron connections and firing patterns and the coarse behavior description has become very large.
But why did neuroscience become so attached to these simplified readouts?
Part of the reason is conceptual. The brain is an overwhelmingly complex system, so one successful strategy was to use an isolationist approach: reduce the problem. Instead of “understand everything the animal is doing,” we can focus on the following question: using one well-defined movement or decision, what can we learn about the specific components of the brain, that are neurons or neuronal circuits?
Part of the reason is technical. Measuring brain activity was only possible when restricting the movement of the animals. Since then, brain imaging and probes were made small enough to be carried by the animal and to allow for free movements. Researchers simply did not have the tools to tackle more complex behavior. It was feasible to count lever presses or spot freezing events by hand; it was not feasible to continuously quantify many subtle behaviors across hours of video.
How AI advances animal behavior research and the challenges ahead
Cameras became cheaper and faster. Deep learning transformed computer vision. It is now possible to follow an animal’s posture in 2D and 3D, at high speed, across long experiments. Tools such as DeepLabCut and LEAP showed that deep neural networks could learn to track user-defined body parts directly from the video recordings. We can finally scrutinize behaviors in varied situations: in their home cage, in the wild, or an experimental arena.
These fast detection methods have been used in countless studies. For example, high-speed tracking has revealed how the cerebellum precisely coordinates paw trajectories and corrects movement errors in rats; applied to facial movements, these methods have shown that subtle expressions are not just visible but encoded across populations of neurons in the cortex, linking behavior directly to internal brain states.
Once you know the pose, the configuration of the animal body at one time, the next question is: how is the body moving through space? One way to approach this is to analyze behavior hierarchically, meaning aggregating poses into short movements, short movements into longer movements, and finally into higher-order actions. Here we find the same concept as in the ethogram, comprehensively describing behavior as small sequences falling into bigger ones. AI can help to detect movement and actions as well as when the action starts and stops. Indeed human experts can recognize events such as grooming, rearing, sniffing, chasing, and AI methods can learn to reproduce what human experts do — but faster. “Annotating every frame in a 1h video with high confidence was estimated to take 22 person-hours” (von Ziegler et al. Neuropharmacology 2021).
AI methods follow two types of approaches: one is based on supervised learning, where the computer directly learns from examples of manually labeled videos; the other is based on unsupervised learning, where similar behavior sequences are grouped together and later on, these groups can be compared to experts’ labels. In mouse models of Alzheimer’s disease, these tools uncovered changes in the structure of behavioral motifs, with more fragmented and disorganized sequences in diseased animals. Large-scale analyses have also challenged long-held assumptions, confirming that female mice are not more variable than males when behavior is described as sequences of actions. Here, AI mainly acts as facilitator: tasks that required hours of human manual annotation the computer now can accomplish much faster.
With the rising amount of collected behavioral data, further models can be built recreating body positions and movement sequences in a virtual engine. These models are established on experimental data recording body poses, muscle forces, bone lengths and angles between them. In these virtual platforms, several components of the movement are simulated and accessible at all times: skeletal, muscular, full body. Systems such as Virtual Rodent or NeuroMechFly equipped with biomechanical constraints (anatomy and body mechanics), sensory inputs and neural commands, can generate movement strategies reminiscent of real animals, and allow to test how body mechanics and neural commands interact to produce behavior. These “digital animals” are powerful as they move beyond passive observation: they let us run experiments that would be difficult, slow, or impossible in vivo, perturbing muscles, neural control, or environmental conditions in a fully controlled setting. Instead of only asking “what did the animal do?”, we can start asking what set of constraints, neural signals, or body mechanics would be sufficient to generate this behavior.
What’s next?
By estimating poses, detecting actions, and building models of behavior, AI represents a major opportunity to scale up: more videos can be processed in a shorter time, more action types can be under scrutiny at once. For all the purposes listed here (animal tracking, pose estimation, action recognition), methods using AI are the one performing the best.
So where is the bottleneck now? I see two.
The first bottleneck is to make tools such as pose estimation and action detection truly accessible and working in different labs. Indeed these tasks are in principle “solved”, methods have been developed and tested. Yet behavioral data vary across laboratories, camera setups, animal strains, experimental protocols… A model that works well in one lab may struggle in another. And this variability remains a challenge for deep learning methods. Sometimes to achieve quality results demands a significant amount of data and expert labels, sometimes the method itself needs to be adjusted to the new case study. In other AI areas, a standard answer to a difficult problem is often: collect more data, train a bigger model, scale up. That strategy works when you have millions of images or billions of words.
And unlike internet-scale AI, animal behavior science has a hard constraint: datasets are often small. In neuroscience and experimental biology, scaling up the data collection is not realistic. Experiments are expensive, slow, involve living animals, and require expertise. We should not assume that more experiments alone will solve the problem.
How do we build more robust methods that work across laboratories? One answer would be to standardize the experiments across labs, buy the same cameras, experimental arenas, … But that would limit our creativity and the development of new experiments to tackle the boundaries of our knowledge. We can build smarter methods by making better use of our data (data augmentation, synthetic data, other types of adjacent data such as time series), by building modular models that can adapt relatively smoothly to new data, and by learning data intrinsic features without human supervision. Making more general models would also mean that we have understood the underlying mechanisms of behavior generation, as variability is a central feature of behavior — not a “bug”: behavior and the underlying neuronal circuits that generate it were selected for adaptation: react to new unseen situations, create different individuals with different bias that favor population survival.
The second bottleneck is learning how to describe behavior in a richer way. Instead of reducing an animal’s actions to a single number, we want to capture how its movements unfold over time. For a long time, this simply was not possible. We lacked both the tools and the theoretical frameworks to describe behavior in such detail. As a result, many current AI methods still do something quite familiar: they replicate what humans were already doing—just faster. But human observation comes with limitations, such as differences between observers, subjective bias, and changes in judgment over time.
To move beyond this, researchers have started to represent behavior not with words, but with numbers. These numerical representations, often called embeddings, place each behavior into a high-dimensional space, a mathematical representation that captures many aspects of a behavior at once. In this space, behaviors that are similar end up close to each other. These embeddings can be built using physics-based models or deep learning approaches, all aiming to better capture the structure of behavioral sequences.
However, this shift brings new challenges. Different methods can produce very different representations, and there is no single “correct” way to group or cluster behaviors. Some behaviors may overlap, and different studies may focus on different levels of detail. Most importantly, validation becomes a central issue: how do we know that these clusters actually correspond to meaningful behaviors?
This raises a deeper question. Is there a universal way to describe behavior—a kind of “behavioral atlas” that works across experiments? Or do we need to choose different tools each time, depending on the question we ask? Exploratory methods are powerful and sensitive, but they also force us to confront a difficult question: what should we trust, and how do we validate what we find?
Understanding behavior requires moving beyond simple measurements toward richer, more dynamic descriptions—and this is where AI becomes essential. While these tools open new possibilities, they also force us to rethink how we interpret and validate what we observe. Ultimately, the goal is not just to analyze behavior more efficiently, but to uncover principles that were previously out of reach.
For further reading
- Von Ziegler, Lukas, Oliver Sturman, and Johannes Bohacek. “Big behavior: challenges and opportunities in a new era of deep behavior profiling.” Neuropsychopharmacology 46.1 (2021): 33-44. https://doi.org/10.1038/s41386-020-0751-7
- Pereira, Talmo D., Joshua W. Shaevitz, and Mala Murthy. “Quantifying behavior to understand the brain.” Nature neuroscience 23.12 (2020): 1537-1549. https://doi.org/10.1038/s41593-020-00734-z
