An international team of biologists and robotic engineers has developed a virtual reality system to decipher how fish school, to help enhance the performance of robotic cars and drones.

The researchers from Germany, Hungary and the US uncovered a natural ‘control law’ used by zebrafish to coordinate behaviour with others in their shoal - a behavioural algorithm that has been fine-tuned over millennia of evolution to facilitate effective collective motion. Aware that understanding such behaviour could be highly advantageous for technological solutions, such as in the control of autonomous vehicles, the scientists tested the performance of the VR system with fleets of robotic cars, drones and watercraft.

The researchers found that the rules of interaction that have evolved in fish are also highly effective in robotic control, demonstrating their potential for the control of robot fleets in the future.

The research, published in Science Robotics, was led by the University of Konstanz and the Max Planck Institute of Animal Behavior (MPI-AB) in Germany, in collaboration with researchers at Eötvös University in Hungary and the Massachusetts Institute of Technology in the US.

Fish are masters of coordinated motion, with individuals staying in formation and avoiding collisions even though they have no leader. They respond with liquid flexibility to changes in their environment, and reproducing this combination of robustness and flexibility has been a long-standing challenge for human engineered systems like robots, say the researchers.

“Our work illustrates that solutions evolved by nature over millennia can inspire robust and efficient control laws in engineered systems,” said Liang Li from the University of Konstanz.

Máté Nagy from Eötvös University underscores this, “The discovery opens up exciting possibilities for future applications in robotics and autonomous vehicle design.”

Nature’s hidden algorithm

Using a VR setup that mimics natural schooling, researchers placed individual juvenile zebrafish into networked arenas where each fish could freely interact with “holographic” virtual members of the same species. Each virtual fish was a projection of a real fish from another arena, meaning that fish could swim and interact together in the same virtual world.

The fully immersive 3D environment lets researchers precisely manipulate visual stimuli and record how the fish respond. This high level of control allowed the scientists to isolate exactly which cues the fish were using to guide their interactions with other fish.

In other words, they could reverse engineer the behaviour of schooling in zebrafish to understand how fish solve the complex problem of coordinating their motion. The solution they discovered was a simple and robust law based only on the perceived position, not the speed, of their neighbours to regulate their following behaviour.

“We were surprised by how little information the fish need to effectively coordinate movements within a school,” says Iain Couzin, senior author on the study and director of MPI-AB and speaker at the Cluster of Excellence Collective Behaviour. “They use local rules that are cognitively minimal, but functionally excellent.”

Real-world testing

To see just how realistic the control law was, the team tested it with real fish. They conducted a VR 'Turing test', based on the concept of testing whether people can tell if they are interacting with a real human or with artificial intelligence. In the aquatic Turing test, a real fish would swim with a virtual fish that switched between being real and being controlled by the algorithm they discovered.

According to the researchers, the real fish could not tell the difference. They behaved the same whether interacting with a real member of the shoal or the virtual follower governed by the algorithm.

To test the broader use of their discovery, the team embedded it in swarms of robotic cars, drones, and boats. The robots were tasked with following a moving target using either parameters from the zebrafish algorithm or from a state-of-the-art method used in autonomous vehicles called model predictive controller (MPC).

Across all tests, the natural control law that fish have evolved delivered performance that was nearly indistinguishable from MPC in terms of accuracy and energy consumption – but at a fraction of the complexity.

Oliver Deussen, a co-author on the study and professor in computer science at the University of Konstanz said, “This work highlights the reciprocal relationship between robotics and biology – using robotics to explore biological mechanisms, which in turn can inspire new and effective robotic control strategies.”