By John Shull, VMASC Assistant Director of Technology for Collaborative Spaces

This project started with an email in the dark to the researchers running the Oyster Hatchery at the Virginia Institute of Marine Science (VIMS). That email came from a simple observation during COVID – that VMASC could do immediate good in our own backyard. We were so focused on these larger problems across the country and the world that in a lot of cases we were passing up immediate opportunities within an area that not everyone has access to – the Chesapeake Bay. So how did I get to this?

During one really nice day in the middle of COVID, I built a tool to help me see how much water I was using when watering my plants. This involved a small Arduino device hooked up to a Flowmeter and it communicated over Bluetooth to my HoloLens. While wearing my HoloLens, I could then display all of this information and augment my physical space with a little graphical interface to let me know how much water each plant had received. This sort of rapid prototyping is what I’ve come to excel at within my job at VMASC and it got me really thinking about what else we could do combining extended reality (XR) tools to everyday problems outside in the environment. I have an odd background of skills and more recently, I generally deal with software development, building XR software (only thing I claim I’m an expert at around here is XR development). I spend a lot of time using game engines to do other things than games – visualizations, simulations, experiences, etc. – and lately I’ve been playing around with a lot more commercial off the shelf (COTS) sensors, things you can buy online from places like Adafruit – they are awesome by the way! I also have gotten pretty good at using existing machine learning and artificial intelligence tools to start to become what I call ‘dangerous’ with them and take advantage of the fact that some of these services have gotten really good at things like identifying objects in an image and it doesn’t take an engineer to reapply these tools to other use cases.

Now back to VIMS. I lucked out and reached Dr. Jessica Moss Small, who is the Director of the Aquaculture Genetics and Breeding Technology Center. She scheduled a zoom call and we realized immediately that we both were very passionate about our areas of interest and expertise and that combined, we could put together a strong team for a lot of advanced applied research opportunities. She connected me with Dr. William Walton (Bill), and we started brainstorming some potential ideas. At the same time, I talked with VMASC administration about getting some internal funding to help upgrade some of our working spaces within the larger lab so we could be even better at prototyping. This project is a split between building up relationships with VIMS and equally expanding some of our fabrication capabilities within the lab. The focus of the COTS sensors is to reduce time to solutions by being able to run machine learning models on device in the field. This is a standard approach that I personally wanted to see VMASC get better at doing. We are focused on a few areas of interest but the running theme between them is applying what I do well and what we can do as a center towards problems in the elements and to help the community within the aquaculture/farming area.

Original Goals

• Goals at the Beginning of the Project: items crossed out are done.
  • Setup infrastructure for in the field IOT data capture and data analysis
  • Setup and extend current VMASC fabrication capabilities within the SeaLab
  • Establish a networking and working relationship with VIMS
  • Extend in the field mixed reality capabilities to Blue Economy problems
  • Extend opportunities for Modeling and Simulation to provide solutions for local communities who rely on the local waterways

Current Status

Below I’m going to list some things I’ve worked on – by no means is this just me doing this; this is knowledge I’ve gained by working with VIMS and some of our own VMASC people and I don’t want to claim that any of this as my accomplishments but more the lab/team accomplishments.

• I’ve established ways of working towards developing custom printed circuit boards (PCB)
  • Anyone at VMASC now has a way to do this through existing University Software and using United States based companies like OshPark.
  • All sensors have been purchased for this project and I have our first custom PCB sent off – should be back in about a week
• Have extensive documentation on sensors and their capabilities for a wide range of environmental sensing purposes
• Established outdoor 3D printing profiles/settings that work within fully submerged conditions and can be exposed to UV for far longer with reducing the potential for waste being left behind
• Have met farmers on the Gulf Coast and identified numerous areas of improvement in which software can help from things like inventory management to better systems for using existing services like RFID tagging all the way up to using drones for gathering information and building custom modular solutions for existing ways of working tied to aquaculture and oyster farming.
• I have identified and am actively building a unique piece of software that will help identify, measure, evaluate, and possibly score a wide range of Oysters from around all US waterways.
  • I am helping build software for VIMS and a program they have underway being led by Dr. Bill Walton that I think will really change how much we know in a given window of time on how Oyster farmers are doing
• Have purchased and tested in the field power management solutions with solar energy and outdoor 4G LTE hotspot hardware that can be utilized for any in-field research operations – this is almost done just waiting on a correct SimCard from AT&T
• ‘Buoy Buddy’ which is my in-house name for this creation/design, is 100% designed by myself
  • For the most part, it’s all 3D printed with a few exceptions for in which those parts are easily found online. The main intention of ‘Buoy Buddy’ is to build a sort of low cost buoy that is really just a massive power bank that contains a small range of COTS sensors that all projects would need
    • These on-board sensors provide local communication protocols (WiFi,BLE), GPS, inertial measurement units that contain gyroscopes and accelerometers, an accurate real-time clock, a water leak detection sensor, and then a large payload area to then house a wide range of other sensors for as needed in the water field research
    • Currently I’ve added an underwater camera, a Ph probe, oxidation-reduction potential (ORP) probe, and a conductivity probe

June 2022 Update

Recently, a lot of the progress we’ve made has come primarily from the collaboration and work with Jordan Lynch and Matthew LaGanke. Jordan Lynch, the Lab & Farm Manager of the Commercial Shellfish Aquaculture Lab, has been helping better understand the new advanced farm they are setting up directly in the James River. I’ve been assisting him in identifying locations for the solar power battery setup that will be helping to power the outdoor LTE hotspot. Due to limitations of existing structures in the water, we are currently testing different configurations on-site at VMASC and will be installing that setup over at VIMS before the end of July. This provides internet access and coverage for up to an additional 100 network-ready devices.

We also have an undergraduate student at Syracuse University helping. Catherine Owens, who is in Industrial and Interaction design, is working on designing a prototype system that will use off-the-shelf equipment to capture data of biofouling in real-time images/video feeds.

With network connectivity out on the farm site, we are building a testbed environment for when future on-edge capabilities advance aquaculture farmers are ready to take complete advantage of that infrastructure. The work with Matthew LaGanke, a graduate student in the Fisheries Sciences, is tied to utilizing RFID/NFC tags to assist in better inventory and farm management. He is working towards finalizing his degree and has been going around and working at various farms within the region to get a good understanding of how technologies like RFID/NFC tags can be utilized to help aquaculture farmers. Currently, he has been testing numerous designs of 3D printed closed encases that can be used in these outdoor settings without getting damaged. There are multiple site tests on-going right now in the Chesapeake Bay and in the Gulf of Mexico to help determine better designs and if these technologies provide that extra added benefit without increasing the costs.