SEAi-MORE: A ShEllfish BehAvior MOnitoRing DEvice

GSI’s team, The Oly Shuckers, was recently selected as a finalist in the OpenCV AI Competition 2021 (#OAK2021) sponsored by OpenCV, Microsoft Azure, and Intel! The proposal was chosen from over 1400 submissions from around the world. The project aims to develop a computer vision system that can provide real-time measurements of in situ shellfish feeding behavior and help to better understand how the feeding behaviors of mussels and oysters change in response to environmental variability.

Background
The U.S. shellfish industry faces domestic and global demands to provide an increasing share of food and raw materials. As the bivalve market continues to grow, it is critical to develop efficient methods to monitor aquaculture stocks to improve organism health and development from seed to market and minimize losses.

Phase 1
Innovative, cost effective, and automated approaches are needed to improve and support aquaculture production systems. The proposed project endeavors to develop a low-cost computer vision-based system to monitor shellfish feeding behavior using OpenCV’s RBG+D camera technology (OAK-D). Project goals are to:

  • Develop a robust, open-source, and low-cost computer vision system to provide real-time measurements of in situ shellfish feeding behavior.
  • Determine how feeding behaviors of mussels and/or oysters raised at commercial aquaculture facilities change in response to environmental variability (e.g., light, temperature, food quantity/quality, pollutants).


Using sensors (e.g., magnetic Hall Effect sensors) to measure valve position as a way of monitoring shellfish feeding behavior is a well-established approach. However, sensor-based approaches require physical manipulation and direct attachment of sensors to the shells of animals, making field deployments challenging and greatly limiting the number of individuals that can be monitored. The Oly Shuckers have proposed an image-based approach to, in large part, streamline and simplify data collection efforts and provide the first ever integrated platform for making simultaneous behavioral and water quality measurements.

Phase 2
Phase 2 will focus on characterizing and field-testing a prototype computer vision system for monitoring shellfish behavior, with an emphasis on collecting imagery that can be used to train robust object detection algorithms (using deep neural networks) that can be deployed on an OAK-D device. At a minimum, the system will include object detection algorithms capable of:

     1. Detecting, segmenting, counting, and estimating basic size/shape metrics for individual bivalves in an image.
     2. Detecting whether valves of individual animals are open, closed, or at some intermediate state.

 A robust training set of imagery suitable for developing neural network-based object detectors for estimating valve-gape status will be an additional deliverable provided as part of this effort. This training set can then be used by others to support future bivalve-related research and development and would represent a substantial advancement for the application of AI in shellfish science.

By simultaneously collecting both imagery and ancillary sensor data The Oly Shuckers will be able to leverage an existing valve-gape measuring device developed by Dr. Sackmann which relies on physical manipulation and attachment of sensors to the shells of individual animals. One benefit of combining our proposed image-based technique with data collected using our existing shell-mounted sensors is that we can streamline the training image annotation process by using sensor data to directly determine valve-gape status.

In a laboratory setting we will also be able to manipulate animals to generate measurable responses to specific environmental perturbations, one at a time, to minimize confounding responses (e.g., change light levels, sound levels, food quality/quantity, salinity). These deliberate manipulations will be conducted to understand how different environmental stressors affect the animals and to generate a more robust image data set that includes a range of animal behaviors for use in training our deep neural networks. Examples of important ancillary water quality measurements that will be collected include:
  • Temperature and salinity
  • Irradiance
  • Sound pressure level
  • Chlorophyll a fluorescence (as an indicator of food quantity)
  • Turbidity
  • Flow/current
After success in a laboratory setting has been achieved, we will attempt to deploy the OAK-D in an underwater housing to capture images of shellfish being cultivated at commercial aquaculture facilities. This will demonstrate that the computer vision algorithms trained with images collected in the lab can be generalized and used to analyze field populations.

The team anticipates that Phase 2 will result in a prototype system capable of monitoring shellfish feeding behavior both in a controlled laboratory setting and in the field. The device will later be extended to simultaneously measure key water quality parameters (e.g., temperature and light intensity). Eventually, all components will be integrated into a small, cost effective, solar-powered platform with a fully integrated telemetry system for real-time reporting.

The Team
Dr. Brandon Sackmann: Is a leader in GSI’s Aquatic Sciences and Environmental Analytics practices. He leverages tools and techniques that bring together large and complex environmental data sets to support strategic and scientifically-based decision making. He has developed comprehensive data management systems and distributed environmental sensor networks and has expertise in marine ecology, satellite remote sensing and ocean optics, computer vision/pattern recognition (CVPR), aquatic plant and algal physiology, and ecosystem modelling.

Brandon recently helped complete a 3-year project (2017 – 2019), funded by the U.S. Department of Energy, to develop an automated system for processing imagery collected with sediment profile imaging (SPI) camera technology. SPI was developed as a reconnaissance tool for characterizing physical, chemical, and biological seafloor processes in near-surface sediments. For SPI applications, Brandon used deep neural networks for image classification, semantic segmentation, and object detection to automatically annotate images and identify infauna (e.g., worms) and key biological features (e.g., feeding voids, burrows). The image processing platform that Brandon developed uses a combination of open-source and commercially available software (e.g., MATLAB and TensorFlow) to streamline and standardize the generation of data from SPI imagery and make image-based data extraction more cost-effective and repeatable.

Dr. Kenia Whitehead: Has a background in biological oceanography. She has developed specific expertise in process level analysis of complex environmental datasets to understand biological responses. Kenia is passionate about the need to communicate scientific results to diverse audiences using effective and interactive data visualizations.

Hannah Podzorski: Has a background in hydrogeology and specializes in the quantitative analysis of complex environmental data sets. She has expertise in data visualization, including the development of customized, interactive dashboards using R/Shiny. The interactive dashboards she creates facilitate complex data exploration by integrating multiple data sets into manageable and interpretable streams of information.

Andy Suhrbier: Is a Senior Biologist with Pacific Shellfish Institute (PSI) in Olympia, WA. Andy helps coordinate PSI's marine benthic/water quality sampling, analysis, and mapping programs. He has also developed shellfish certification standards for the west coast shellfish industry. Andy’s current projects and duties include: the development of sea cucumber aquaculture in upland and marine areas, tracking mobile fauna inside and outside aquaculture, shellfish bed population estimates and mapping, shellfish and kelp aquaculture farm permitting, and maintaining water quality monitoring stations related to ocean acidification, a part of the Northwest Association of Networked Ocean Observing Systems (NANOOS). He enjoys interacting with shellfish producers and researchers in many growing areas throughout Alaska, California, Oregon, and Washington.

PSI facilities include a well-equipped experimental laboratory space. A bank of 5-gallon aquariums with associated air bubblers, nets, and substrate are available for wet lab experiments. PSI also houses multiple instruments at this location and out in the field, from multiparameter sondes (YSI EXO II) to temperature recorders (HOBO TidbiT®).