Researchers use machine learning to listen for specific fish sounds

Come mating season, fishes off the California coast sing songs of love in the evenings and before sunrise. They vocalize not so much as lone crooners but in choruses, in some cases loud enough to be heard from land. It’s a technique of romance shared by frogs, insects, whales, and other animals when the time is right.

For most of these vocal arrangements, the choruses are low-frequency. They’re hard to distinguish from the sounds of ships passing in the night among others.

Biologists, however, have long been interested in listening in on them in the name of understanding fish behavior toward an ultimate goal: They can help preserve fish populations and ocean health by identifying spawning seasons and using that to inform fisheries management.

Now scientists at Scripps Institution of Oceanography at UC San Diego and colleagues have developed a way for computers to sift through sounds collected by field acoustic recording packages and process them faster than even the most trained human analysts. The method could represent a major advance in the field of signal processing with uses beyond marine environments.

“My hope is that this method will help others to automatically separate and detect signals more easily in order to understand where and when fish are mating,” said Ella Kim, a Scripps Oceanography graduate student and lead author of the study. “Beyond fish, this method could easily be applied to other soundscapes to learn more about frogs, birds, bats, and other animals.”

In the NOAA-U.S. Navy-funded study, Kim and colleagues separated fish chorusing sounds from the overall soundscape using a technique she calls SoundScape Learning (SSL). SoundScape Learning combines established methods of sound analysis with a neural network, an information processing paradigm based on how neurons in the human brain enable cognition.

Study co-author Kaitlin Frasier, a Scripps Oceanography marine acoustician who has used artificial intelligence in her own research analyzing dolphin clicks, said scientists often train machines to look for specific sounds in the ocean, but the magnitude of the soundscape makes that a difficult task.

“Ella’s method aims to teach computers to look at the bigger picture, finding and learning to recognize sound events, like taking in the forests instead of the trees.”

The “denoising,” as researchers call it, comes in as SSL separates events that occur over and over again and puts them in what researchers term a “low-rank” matrix. Those steady sounds are separate from transient sounds, such as those made as whales or ships pass by. Fish choruses take place every night so they are placed by the program into the low-rank matrix. The transient sounds are sorted into a “sparse” matrix.

Kim and Frasier offer various metaphors to help non-acousticians get the concept. Kim suggests thinking of a traffic camera recording footage of a stretch of freeway for hours on end. The road itself would be in the low-rank matrix. The video of cars driving on it gets sent to the sparse matrix.

Frasier offers, “Instead of trying to recognize individual words in a recording, it’s like we put a mic on a street corner, and we’re now able to identify small groups of people walking by and chatting, a plane flying over, and light vehicle traffic. The algorithm is learning to identify the acoustic hallmarks of each of those categories on its own, by reviewing lots of data.”

The time savings is significant, said Kim. In the study, the machines listened to 5.3 years of audio gathered off the California coast. Were a skilled acoustician knowledgeable on fish chorusing in the area to do this work manually, it would probably take them a month, but once SSL is trained, the process takes only a matter of days.

The study appeared March 15, 2023 in The Journal of the Acoustical Society of America. Co-authors include Annebelle Kok, Gabrielle Arrieta, Sean Wiggins, and Simone Baumann-Pickering of Scripps Oceanography; Megan McKenna and Lindsey Peavey Reeves of the National Marine Sanctuary Foundation; and William Oestreich of Monterey Bay Aquarium Research Institute.